Research map / Large language models / Language, safety and interpretability
UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace temperature sampling with capped, approximately uniform UniMax sampling | mT5-style multilingual pretraining · TyDi QA GoldP, Large/XL/XXL | average EM/F1: 82.0 -> 83.3; 83.4 -> 83.9; 84.0 -> 84.4 | Helped |
| Replace temperature sampling with UniMax sampling | umT5 pretraining · XNLI translate-train, XXL | accuracy: 87.1 -> 87.8 | Helped |
| Replace temperature sampling with UniMax sampling | umT5 pretraining · PAWS-X, XXL | accuracy: 91.5 -> 91.2 | Hurt |
| Replace temperature sampling with UniMax sampling | umT5 pretraining · XQuAD, XXL | EM: 71.3 -> 77.9 | Helped |
| Replace temperature sampling with UniMax sampling | umT5 pretraining · MLQA, XXL | EM: 58.3 -> 70.5 | Helped |
| Replace temperature sampling with UniMax sampling | umT5 pretraining · TyDi QA, XXL | EM: 79.5 -> 81.2 | Helped |
| Increase UniMax training budget to 1/2 character budget (ablation) | Large UniMax · TyDi QA | average score: 82.2 -> 83.1 | Helped |
| Allow repeats up to 5 epochs instead of N=1 (ablation) | UniMax · TyDi QA, Large | average score: 82.2 -> 81.5 | Hurt |
| Allow repeats up to 10 epochs instead of N=1 (ablation) | UniMax · TyDi QA, Large | average score: 82.2 -> 81.8 | Hurt |
| Refresh the mC4 corpus (ablation) | UniMax-trained model · XNLI translate-train | accuracy: 77.6 -> 77.7 (+0.1) | No clear effect |
Closest papers by meaning
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.88)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.88)
- Improving Diversity in Language Models: When Temperature Fails, Change the Loss (ICML 2025 · similarity 0.88)
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining (NeurIPS 2025 · similarity 0.88)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.87)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.