Research map / Optimization / Optimizers
Fast Data Mixture Optimization via Gradient Descent
Haoru Tan, Sitong Wu, Yanfeng Chen, Jun Xia, Ruobing Xie, Bin Xia, Xingwu Sun, Xiaojuan Qi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace many proxy models (RegMix 512, CLIMB 64) with a single proxy trained by gradient-based mixture updates | FastMix · Pile pre-training | average downstream score: 47.2 -> 48.2 vs RegMix; 47.5 vs CLIMB | Improved |
| use single proxy gradient-based search | FastMix · Pile pre-training | search cost: 720.5 -> 1.3 GPU-hours vs RegMix | Improved |
| use FastMix mixture for SFT of Qwen2.5-Math-Instruct 7B | FastMix · SFT math/code/STEM | average score: 59.9 -> 65.4 vs CLIMB | Improved |
| use outer-loop horizon n2 greater than 1 (ablation) | FastMix · industrial data | stability and cost | Hurt |
| use black-box gradient estimators (finite differences, SPSA) for non-differentiable metrics (ablation) | FastMix · industrial data | convergence | Hurt |
| add entropy regularization to the mixture (ablation) | FastMix · industrial data | mixture quality | No clear effect |
| use small proxy models (below 0.5B) on industrial data (ablation) | FastMix · industrial data | mixture ratio stability | Hurt |
| concatenate short SFT sequences to match pre-training length when using SFT loss as target (ablation) | FastMix · pre-training with SFT-loss target | gradient alignment | Improved |
Closest papers by meaning
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.88)
- RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.88)
- MixMin: Finding Data Mixtures via Convex Minimization (ICML 2025 · similarity 0.87)
- Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (ICML 2026 · similarity 0.86)
- CheMixHub: Datasets and Benchmarks for Chemical Mixture Property Prediction (NeurIPS 2025 · similarity 0.85)
- A Biconvex Formulation for Stable Transport of Mixture Models with a Unique Solution (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
