Research map / Large language models / Instruction and fine-tuning
Fine-Tuning Language Models with Collaborative and Semantic Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoE, a two-phase SFT method that trains domain-specific FFN experts (general, math, coding) on LLaMA2-7B and then fuses them into a semantically routed top-2 MoE with a margin routing loss.
Problem
Balancing general knowledge, coding, and math capabilities in LLMs during supervised fine-tuning, where mixing domain data causes capability loss and MoE experts lack interpretable specialization.
Approach
Train FFN-only experts on capability-labeled SFT subsets (labels from GPT-4 prompting), then freeze the experts and combine them into a MoE with a router trained under a margin loss that pushes routing toward the semantically labeled expert.
Key results
- CoE-3E2A reaches 50.47 MMLU, 44.73 GSM8K CoT, 26.22 HumanEval, and 44.81 GSM8K PoT, versus 49.91/40.33/19.51/31.69 for LLaMA2-7B-SFT on the same suite.
- CoE-3E2A scores 6.37 on MT-Bench and 73.01% win rate on AlpacaEval, versus 5.91 and 62.69 for the SFT baseline.
- Removing the margin routing loss drops the average from 40.47 to 38.54.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoE-3E2A rather than LLaMA2-7B-SFT | LLaMA2-7B-Base fine-tuning · GSM8K PoT, 3-shot | execution accuracy: 31.69 -> 44.81 | Helped |
| Update all parameters during expert leveraging rather than use FFN-excluded fine-tuning (ablation) | CoE-3E2A · MT-Bench Overall | score: 6.37 -> 5.95 | Hurt |
| Update all parameters during expert leveraging rather than use FFN-excluded fine-tuning (ablation) | CoE-3E2A · AlpacaEval | win rate: 73.01 -> 70.20 | Hurt |
| Update only router modules during expert leveraging rather than use FFN-excluded fine-tuning (ablation) | CoE-3E2A · GSM8K PoT, 3-shot | execution accuracy: 44.81 -> 33.43 | Hurt |
| Replace trained experts with base-model duplicates (ablation) | CoE-3E2A · GSM8K PoT, 3-shot | execution accuracy: 44.81 -> 32.37 | Hurt |
| Replace trained experts with base-model duplicates (ablation) | CoE-3E2A · AlpacaEval | win rate: 73.01 -> 67.60 | Hurt |
| Match activated parameters with a two-expert MoE instead of using CoE-3E2A (ablation) | CoE-3E2A · GSM8K PoT, 3-shot | execution accuracy: 44.81 -> 33.81 | Hurt |
| Remove margin routing loss (ablation) | CoE-3E2A · MMLU | accuracy: 50.47 -> 50.06 | Hurt |
| Remove margin routing loss (ablation) | CoE-3E2A · GSM8K | accuracy: 44.73 -> 41.17 | Hurt |
| Remove margin routing loss (ablation) | CoE-3E2A · HumanEval | pass rate: 26.22 -> 24.39 | Hurt |
Surprising findings
- Mixing all SFT data (LLaMA2-7B-SFT) underperforms the single-domain expert models on their own specialty benchmarks, suggesting capability conflict during SFT.
- Fully updating all parameters of the MoE (CoE-3E2A-full) slightly degrades performance relative to freezing experts.
Limitations
- Evaluated only on LLaMA2-7B base with three capability groups.
- Labels for capability groups depend on GPT-4 prompting and no label quality analysis is reported.
- Routing interpretability is shown qualitatively on a single example.
Experimental setup
Datasets: TULU-v2 ShareGPT WizardLM FLAN Open-Orca GPT4-Alpaca Open Assistant 1 CodeAlpaca OSS-Instruct MAmmoTH CoT
Benchmarks: MMLU GSM8K HumanEval MT-Bench AlpacaEval
Metrics: accuracy pass@1 MT-Bench score win rate
Baselines: LLaMA2-7B-SFT Expert-General Expert-Math Expert-Coding MoE-3E2A MoE-2E2A CoE-3E2A-full CoE-3E2A-router
Builds on: Mixture-of-Experts Branch-Train-Mix LLaMA2 Lemur Program-of-Thought
Closest papers by meaning
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.89)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.89)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.89)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.88)
- Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.88)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.