Research map / Large language models / Instruction and fine-tuning

TANDEM: Bi-Level Data Mixture Optimization with Twin Networks

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TANDEM, a penalized single-level reformulation of bi-level data mixture optimization solved with synchronized proxy and reference "twin" networks, with a convergence guarantee and experiments in data-abundant, data-restricted, and SFT settings.

Problem

Finding optimal domain mixture ratios for LLM training data is a bi-level optimization problem that existing methods (DoReMi, DoGE, Skill-It, Aioli) handle poorly, especially in data-restricted and supervised fine-tuning settings.

Approach

Relaxes the inner training constraint into a Lagrangian penalty and solves the resulting single-level problem with a proxy model trained on training data and a reference model trained on training plus validation data, synchronized at each mixture update; domains whose loss gap between the two models is larger are up-weighted.

Key results

• In the data-restricted pretraining setting, TANDEM reaches 28.07 average perplexity on 160M models, beating the best baseline Skill-It (29.24) and Uniform (31.53).
• On the 500M Qwen2 SFT mixture, TANDEM attains the lowest test loss (0.208 vs 0.231 for Uniform) and highest average metric (83.3).
• In the data-abundant 160M setting, TANDEM (25.43 average perplexity) is competitive with Uniform (25.74) and Aioli (25.66) rather than clearly superior.

Measured effects reported in the paper

Surprising findings

• In the data-abundant one-epoch setting, uniform mixing remains highly competitive and many DMO methods fail to consistently beat it.
• DoGE shows the largest variance in its hyper-gradient estimate in the SFT setting, and increasing probing steps K reduces variance.

Limitations

• Convergence guarantee relies on several strong assumptions (PL condition, bounded Hessian, Lipschitz smoothness) and yields only a first-order stationarity rate of O(T^-1/4).
• Data-abundant results are not clearly better than the uniform baseline.
• Experiments are limited to small GPT-style models (160M to 1B) and one 500M SFT model.

Experimental setup

Datasets: SlimPajama Natural Instructions

Benchmarks: SlimPajama per-domain perplexity Natural Instructions SFT tasks

Metrics: perplexity average perplexity test loss accuracy

Baselines: Uniform DoReMi DoGE Skill-It Aioli

Builds on: DoReMi DoGE Skill-It Aioli

Closest papers by meaning

• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.89)
• Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (ICML 2026 · similarity 0.88)
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.88)
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.87)
• Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning (ICML 2025 · similarity 0.87)
• LLM Data Selection and Utilization via Dynamic Bi-level Optimization (ICML 2025 · similarity 0.87)