Research map / Large language models / Language, safety and interpretability
Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIMB, a scaling-law framework with a cross-lingual interaction-aware language ratio and a two-step optimizer to pick multilingual pretraining language proportions, validated with 1.2B and 7B models trained on 1T tokens of FineWeb-2.
Problem
Choosing optimal language proportions in multilingual LLM pretraining is hard because languages interact through cross-lingual transfer and optimal ratios shift with data scale, while retraining for each mixture is prohibitively expensive.
Approach
Defines a cross-lingual interaction-aware language ratio (the monolingual token budget equivalent to a language's multilingual validation loss, divided by total budget), models it parametrically with pairwise transfer terms inside a per-language scaling law, and finds allocations via a two-step procedure: marginal-benefit balancing for direction, then constrained magnitude maximization.
Key results
- On the 2-language setting at 100B tokens, CLIMB's scaling law reaches R^2 0.978 with Huber loss 0.518e-3, versus 0.832 / 5.61e-3 for MSL.
- The 1.2B CLIMB-allocated model outperforms Uniform, Isolated, Natural and MSL allocations by about 1.85% on average across benchmarks, with up to 2.60% gain on XNLI over isolated allocation.
- CLIMB allocations at 7B scale outperform baselines by an average of 3.4%, and the 1.2B model is competitive with LLaMA-3.2, Gemma-3 and Qwen-3 despite training on only 1T tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIMB-derived language allocation instead of alternative allocation strategies | LLaMA-2 1.2B · Multilingual benchmarks (XNLI, XCOPA, XStoryCloze, FLORES, GMMLU, JMMLU, VMLU, XWinograd, XHS, XARC-E, XARC-C, XGPQA, XTQA) | benchmark score | Mixed |
| Use the cross-lingual interaction-aware scaling law instead of MSL | Multilingual validation-loss prediction · 2-language LLM, 100B tokens | R²: 0.832 -> 0.978 | Helped |
| Use the cross-lingual interaction-aware scaling law instead of MSL | Multilingual validation-loss prediction · 2-language LLM, 100B tokens | Huber loss (× 10^-3): 5.61 -> 0.518 | Helped |
| Increase training data budget above 25B tokens | CLIMB scaling-law prediction · Scaling-law applicability | predictive accuracy | Helped |
| Increase token budget from smaller scales to larger scales | Cross-lingual interaction-aware ratio · Multilingual pretraining | cross-lingual transfer strength | Hurt |
Surprising findings
- Optimal language allocations shift with data scale: low-resource or simpler languages get higher share at small budgets, while complex languages gain share as data grows.
- Cross-lingual transfer coefficients can be negative (interference), and transfer strength is non-monotonic in the token budget.
Limitations
- Only handles languages already present in the training corpus; cannot predict for unseen languages.
- Scaling-law predictions are unreliable below about 25B tokens due to unstable cross-lingual transfer.
- Translated benchmarks (XHS, XARC, XGPQA, XTQA) were produced via GPT-based translation, and scale of experiments limited to 1.2B and 7B models.
Experimental setup
Datasets: FineWeb-2
Benchmarks: XNLI XCOPA XStoryCloze Flores GMMLU CMMLU JMMLU VMLU XWinograd XHS XARC-E XARC-C
Metrics: R^2 Huber loss accuracy chrF++
Baselines: Isolated MSL Uniform Natural proportions LLaMA-3.2-1B Gemma-3-1B-pt Qwen-3-1.7B-base
Builds on: Chinchilla scaling laws multilingual scaling law (MSL) DoReMi
Closest papers by meaning
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.89)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.89)
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.88)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.88)
- The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model (ICLR 2025 · similarity 0.88)
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.