Research map / Architectures and efficiency / Mixture of experts
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Soup-of-Experts, a pretraining architecture whose expert parameters are linearly combined with coefficients predicted from domain weights, so a specialist small model can be instantiated instantly for any domain mixture.
Problem
Obtaining small specialist language models for scarce target domains normally requires retraining a model per specialization dataset (e.g., importance-sampling/CRISP), which scales poorly with the number of downstream tasks.
Approach
Pretrain a large bank of expert weights plus shared parameters, mapping domain weights through a small MLP to linear combination coefficients; train by sampling random domain weights from a meta-distribution and backpropagating through a batch from the corresponding mixture, so any specialist model is instantiated by parameter averaging without retraining.
Key results
- Soup-of-Experts (128 experts, 110M instantiated models) achieves the best average specialized loss across 16 Pile domains, outperforming generic pretraining, domain experts and CRISP.
- Soup-of-Experts is only slightly worse than generic pretraining on the RedPajamaV2 generic loss, indicating it retains general knowledge.
- On uspto, fine-tuning the generic model needs about 10M tokens to match the out-of-the-box Soup-of-Experts loss; only about three specialization samples are needed to match the generic model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Soup-of-Experts instead of generic pretraining | GPT-2 110M · 16 Pile specialization domains | next-token prediction loss | Helped |
| use Soup-of-Experts instead of generic pretraining | GPT-2 110M · RedPajamaV2 pretraining distribution | next-token prediction loss | Hurt |
| use Soup-of-Experts instead of CRISP or domain experts | GPT-2 110M · Main experiments | training throughput relative to generic pretraining | Hurt |
| increase instantiated model size (ablation) | Soup-of-Experts · Specialized Pile domains across three model scales | next-token prediction loss | Helped |
| use three samples to estimate specialist domain weights (ablation) | Soup-of-Experts · Specialized Pile domains | next-token prediction loss | Helped |
| use 100–1000 samples to estimate specialist domain weights (ablation) | Soup-of-Experts · Specialized Pile domains | next-token prediction loss | Helped |
| set meta-distribution support size to 1 (ablation) | Soup-of-Experts · Specialized and generic datasets | next-token prediction loss | Hurt |
| increase meta-distribution support size (ablation) | Soup-of-Experts · Generic dataset | next-token prediction loss | Helped |
| set meta-distribution support size to 2 (ablation) | Soup-of-Experts · Specialized datasets | next-token prediction loss | Helped |
| match training support size to test mixture support size (ablation) | Soup-of-Experts · Random mixtures with fixed support size | next-token prediction loss | Helped |
| use low-rank experts instead of dense experts (ablation) | Soup-of-Experts · Specialized datasets | next-token prediction loss | Hurt |
| vary number of low-rank experts at fixed total parameter count (ablation) | Low-rank Soup-of-Experts · Specialized datasets | next-token prediction loss | No clear effect |
Surprising findings
- Low-rank experts are less parameter-efficient than dense experts at a fixed parameter count, contrary to the expectation that more experts would help.
- Best specialized loss is obtained with meta-distribution support size s=2 rather than larger supports.
Limitations
- Specialization evaluated only with next-token loss, not downstream task performance.
- Low-rank experts underperform dense experts.
- Training cost and total parameter count grow with the number of experts (e.g., 14B parameters for 128 experts of 110M).
Experimental setup
Datasets: RedPajamaV2 The Pile (16 specialization domains)
Benchmarks: The Pile (16 domains) next-token loss RedPajamaV2 generic loss
Metrics: next-token prediction loss average specialized loss generic loss
Baselines: Generic pretraining Domain experts CRISP
Builds on: CRISP Domain experts Task arithmetic / model soups LoRA Sentence-BERT
Closest papers by meaning
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Autonomy-of-Experts Models (ICML 2025 · similarity 0.87)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.87)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.87)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.87)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.