Research map / Optimization / Optimizers

MixMin: Finding Data Mixtures via Convex Minimization

ICML 2025 · Optimization › Optimizers · approach: Surrogate losses and robust solvers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formulates data mixing as a bi-level objective that reduces to convex minimization for cross-entropy/MSE with large model classes, and proposes MixMin, an entropic-descent method over cheap proxy models' predictions, improving pythia-410M and XGBoost data mixtures.

Problem

Choosing mixture weights over data sources to minimize a downstream loss is a bi-level optimization that is generally intractable and usually requires expensive grid or random search.

Approach

With CE or MSE loss and no covariate shift, the bi-level data mixing objective becomes a convex problem over mixtures of the Bayes-optimal source models; MixMin approximates this with cheap per-source proxy models and optimizes the mixture weights via entropic mirror descent on downstream samples.

Key results

• MixMin with proxy compute of 1% of the 160M training run improved generative loss over the natural mixture on all four target tasks, with 1-5% relative NLL improvement for pythia-410M trained on 8.2B tokens.
• MixMin mixtures found with 160M proxies improved pythia-410M training on most tasks as scale increased, without refitting weights.
• On PCBA chemistry, MixMin improved average precision by about 0.03-0.15 over training on the target alone as the number of surrogate assays grew.

Measured effects reported in the paper

Surprising findings

• Ensembling the proxy models with MixMin weights performed much worse than retraining on the MixMin mixture, implying proxies were far from Bayes optimal yet still gave good mixtures.
• MixMin performance did not degrade substantially when proxy models were trained with as little as 1% of full training compute.

Limitations

• Theory applies only to cross-entropy or MSE losses without covariate shift among sources, excluding e.g. image classification.
• The convex reduction relies on large model classes; proxy models are far from Bayes optimal in practice.
• How cheap proxy models can be without degrading performance remains open.

Experimental setup

Datasets: SlimPajama PIQA ARC Easy SciQ OpenWebMath PCBA (PubChem bioassays)

Benchmarks: PIQA ARC Easy SciQ OpenWebMath PCBA

Metrics: negative log likelihood (generative loss) predictive loss accuracy average precision

Baselines: RegMix random search balanced mixture natural mixture

Builds on: RegMix mirror descent Chinchilla scaling

Closest papers by meaning

• Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (ICML 2026 · similarity 0.88)
• Aioli: A Unified Optimization Framework for Language Model Data Mixing (ICLR 2025 · similarity 0.88)
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.88)
• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.87)
• RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.87)
• Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.86)