Research map / Label-efficient and robust learning / Domain adaptation and generalization
Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Chameleon, a data-mixing method that computes domain weights from kernel ridge leverage scores over proxy-model domain embeddings, enabling cheap pretraining mixtures, transfer to new domains without proxy retraining, and finetuning mixtures.
Problem
Domain reweighting for LLM pretraining and finetuning is costly because existing methods (DoReMi, DoGE) need proxy-model optimization and retraining whenever domains change, and they are mostly limited to pretraining.
Approach
Build a linear-kernel domain affinity matrix from mean proxy-model hidden embeddings per domain and use kernel ridge leverage scores as domain importance: inverse KRLS (Christoffel-like density) for pretraining weights, and KRLS directly for finetuning weights, with no proxy optimization loop.
Key results
- On SlimPajama with 684M models, Chameleon reaches average PPL 22.31 versus 24.20 for Uniform, 23.34 for DoReMi and 22.32 for DoGE, at about 1.36e17 FLOPs versus 6.68e17 for DoGE and 1.34e18 for DoReMi.
- On transfer to the Pile, Chameleon achieves average PPL 19.94 versus 23.05 for the Human mixture and 30.17 for RegMix, at 4.62e15 extra FLOPs (about 1/145 of DoGE's cost).
- In finetuning on Stack programming languages, Chameleon lowers average PPL from 22.02 (Uniform) to 17.18 across all 7 languages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use inverse KRLS domain weights instead of uniform mixing for pretraining | Chameleon · SlimPajama-627B test domains | Average PPL: 24.20 -> 22.31 | Helped |
| Use inverse KRLS domain weights instead of uniform mixing for pretraining | Chameleon · SlimPajama downstream tasks | Average accuracy: 37.9 -> 39.6 | Helped |
| Use inverse KRLS domain weights instead of uniform mixing for pretraining | Chameleon · SlimPajama-627B test domains, 1.2B model | Average PPL: 16.86 -> 15.03 | Helped |
| Use inverse KRLS domain weights instead of uniform mixing for pretraining | Chameleon · SlimPajama downstream tasks, 1.2B model | Average accuracy: 40.5 -> 41.5 | Helped |
| Transfer weights to new domains using the existing proxy rather than retraining a proxy | Chameleon · Pile new domains | Average PPL: 23.05 -> 19.94 | Helped |
| Transfer weights to new domains using the existing proxy rather than retraining a proxy | Chameleon · Pile downstream tasks | Average accuracy: 38.2 -> 39.7 | Helped |
| Use KRLS weights rather than uniform domain weights for fine-tuning | Chameleon · Wiki40b, 7 Latin languages | Average PPL: 9.43 -> 8.58 | Helped |
| Use KRLS weights rather than uniform domain weights for fine-tuning | Chameleon · Stack dataset, 7 programming languages | Average PPL: 22.02 -> 17.18 | Helped |
| Use KRLS weights rather than inverse KRLS weights for fine-tuning (ablation) | Chameleon · Wiki40b and Stack fine-tuning | test perplexity | Helped |
| Derive domain weights from a proxy trained for 2k steps (ablation) | Chameleon · SlimPajama-627B test domains | Average PPL: 26.34 -> 22.31 | Helped |
Surprising findings
- Inverse KRLS weights (alpha^PT) for finetuning performed worse than KRLS weights, supporting phase-specific weighting.
- RegMix, which uses downstream task knowledge, has higher average perplexity than the data-agnostic Chameleon on both SlimPajama and Pile.
Limitations
- Pretraining weights rely on inverse KRLS with a softmax temperature that is tuned per phase (tau^PT in [5,10], tau^FT in [0.2,0.5]).
- Evaluated mainly on 7-domain SlimPajama and a 17-domain Pile subset; domain-level only, no sample-level mixing.
- Downstream results show small absolute gains with differences often within noise.
Experimental setup
Datasets: SlimPajama-627B Pile Wiki40b Stack-dedup
Benchmarks: SlimPajama per-domain perplexity Pile per-domain perplexity ARC-E COPA HellaSwag Lambada LogiQA MultiRC OpenBookQA PiQA QQP RACE
Metrics: perplexity accuracy GPU hours FLOPs
Baselines: Uniform DoReMi DoGE RegMix Data Mixing Laws Human (Pile)
Builds on: DoReMi DoGE RegMix kernel ridge regression leverage scores
Closest papers by meaning
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
- DOGE: Domain Reweighting with Generalization Estimation (ICML 2024 · similarity 0.89)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.88)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.88)
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.88)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.