Research map / Architectures and efficiency / Mixture of experts
Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
Jingwei Li, Xinran Gu, Jingzhao Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| model the validation loss with a capacity-aware joint law of model size and mixture (CAMEL) instead of separable laws | CAMEL vs DML/SODM · 7 benchmarks, Balanced objective, 55B-A1.2B | weighted average score: 0.526 -> 0.532 (vs SODM); 0.519 (DML) | Improved |
| use CAMEL mixture instead of human-designed mixture | CAMEL · Balanced objective weighted average | weighted average score: 0.496 -> 0.532 | Improved |
| use CAMEL mixture instead of model-size-agnostic mixture | CAMEL · Balanced objective weighted average | weighted average score: 0.514 -> 0.532 | Improved |
| use CAMEL for Math specialized target | CAMEL vs SODM · Math specialized weighted average | weighted average score: 0.494 -> 0.497 | Improved |
| use CAMEL for Code specialized target | CAMEL vs SODM · Code specialized weighted average | weighted average score: 0.520 -> 0.528 | Improved |
| use CAMEL for Knowledge specialized target | CAMEL vs SODM · Knowledge specialized weighted average | weighted average score: 0.559 -> 0.565 | Improved |
| use hourglass sampling of (model size, mixture) pairs instead of rectangle allocation (ablation) | CAMEL fitting · extrapolation error under fixed compute | prediction error | Improved |
| compare triangle, inverted triangle and diamond sampling against hourglass (ablation) | CAMEL fitting · extrapolation error under fixed compute | prediction error | Hurt |
| vary the number of intrinsic domains k (ablation) | CAMEL · loss prediction | prediction error | No clear effect |
| use CAMEL instead of DML on MATH | CAMEL vs DML · MATH (Balanced mixture) | accuracy: 0.393 -> 0.372 | Hurt |
| use CAMEL instead of SODM on GSM8K | CAMEL vs SODM · GSM8K (Balanced mixture) | accuracy: 0.705 -> 0.693 | Hurt |
| fit loss-to-benchmark logistic law on validation losses | loss-to-benchmark law · 14 benchmarks | benchmark prediction error | Improved |
Closest papers by meaning
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.91)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.89)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.88)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.88)
- $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.88)
- RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
