Research map / Large language models / Language, safety and interpretability

Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Self-MoE, which turns a monolithic LLM into MiXSE, a mixture of lightweight LoRA experts built from self-generated synthetic data per domain, combined by a router trained on that data.

Problem

Specializing LLMs for target domains usually needs costly human-labeled data and tends to degrade performance on non-target tasks (forgetting); merging or MoE alternatives are static or need extra trained modules or large pretraining.

Approach

Self-specialize a base LLM into separate LoRA experts for knowledge, reasoning, math, and coding using 100 seeds per domain and self-generated data (about 20K examples total), then train only a shared top-k router on the aggregated data so experts stay semantically distinct while the base weights remain active.

Key results

• MiXSE (Top-1) averages 54.3 across MMLU, BBH, GSM8K, HumanEval vs 47.8 for Gemma-7B base, a 6.5-point average gain.
• Instance Merging averages 52.4 and TIES/DARE merging average 47.9 and 44.3 on the same four benchmarks.
• Knowledge self-specialized expert improves MMLU to 64.0 but drops reasoning, math, and coding, illustrating specialization trade-offs.

Measured effects reported in the paper

Surprising findings

• Single-domain self-specialized experts degrade non-target domains (e.g., Knowledge Self-Spec. drops BBH from 56.1 to 41.7).
• Seed-only training gives only marginal gains, while MiXSE yields larger gains, indicating the gains are not from seed data alone.

Limitations

• Synthetic data may raise contamination and noise concerns; only n-gram overlap checks were done.
• Compute constraints meant the model and data were not scaled to full potential; XLoRA MoE implementation was not optimized.
• MiXSE still trails Mixtral 8x7B on the reported benchmarks.

Experimental setup

Datasets: MMLU BBH GSM8K MBPP self-generated synthetic data

Benchmarks: MMLU BBH GSM8K HumanEval MATH MBPP Natural Questions TriviaQA Hellaswag PIQA TruthfulQA

Metrics: accuracy average accuracy across domains

Baselines: Self-specialized single-domain models Instance Merging (multi-task tuning) TIES merging DARE merging BTX Sparse Upcycling BTM GLAN Orca-2 Merlinite Mixtral 8x7B

Builds on: self-specialization LoRA Top-k MoE routing X-LoRA TIES DARE

Closest papers by meaning

• Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.90)
• LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.89)
• OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.89)
• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.88)
• Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration (NeurIPS 2025 · similarity 0.88)
• Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.87)