Research map / Large language models / Language, safety and interpretability
MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MuRating, which distills an ensemble of English data-quality raters into a pairwise Bradley-Terry scorer and transfers it via translation to 17 languages (monolingual, cross-lingual, parallel pairs) to select multilingual pretraining data.
Problem
Model-based data-quality selection for LLM pretraining has been developed almost entirely for English; multilingual data selection lacks reliable, contamination-free quality raters.
Approach
Aggregate several English raters via majority-vote pairwise preferences fit with a Bradley-Terry loss, then translate scored English pairs into 17 languages, adding cross-lingual pairs and neutral-label parallel pairs so a single multilingual BGE-M3-based rater scores documents language-agnostically.
Key results
- On 12 English benchmarks with 1.2B models trained on 200B tokens, MuRater averages 51.23 vs 50.23 for DCLM and 48.70 for uniform sampling with 50% more data.
- On multilingual benchmarks (18 languages, 200B English + 300B multilingual tokens), MuRater(E) averages 50.96 vs 49.19 for QuRater-M in the category-averaged table; on the 13-language subset it reaches 51.81 vs 48.97 for FineWeb2-HQ.
- At 7B scale on 1T tokens with 16.5% multilingual data, MuRater averages 52.61 vs 51.54 for QuRater-M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select English pretraining data with MuRater instead of the strongest reported baseline, DCLM | LLaMA 1.2B pretraining · English benchmarks (12-task average) | average: 50.23 -> 51.23 | Helped |
| select multilingual pretraining data with MuRater(E) instead of QuRater-M | LLaMA 1.2B pretraining · Multilingual benchmarks, 18 languages (11-task average) | average: 49.19 -> 50.96 | Helped |
| use English-rated pairs translated into multilingual pairs (MuRater(E)) instead of multilingual pairs translated into English (MuRater(M)) | MuRater multilingual data selection · Multilingual benchmarks, 18 languages (11-task average) | average: 50.02 -> 50.96 | Helped |
| select multilingual pretraining data with MuRater(E) instead of FineWeb2-HQ | LLaMA 1.2B pretraining · Multilingual benchmarks, 13-language subset (11-task average) | average: 48.97 -> 51.81 | Helped |
| select multilingual pretraining data with MuRater instead of QuRater-M | LLaMA 7B pretraining on 1T tokens · Multilingual benchmarks (average) | average: 56.18 -> 57.23 | Helped |
| add cross-lingual and parallel pairs to rater training (ablation) | MuRater multilingual scoring · Parallel multilingual documents | cross-lingual score consistency (MSE and slope) | Helped |
| transfer quality judgments with pairwise comparisons rather than pointwise scores (ablation) | English-to-multilingual quality scoring · Arabic and Spanish translated pairs | cross-lingual score consistency | Helped |
| increase the confidence margin from 50% to 80% (ablation) | MuRater validation predictions · Validation set (held-out) | accuracy: 90.7% -> 93.1% | Helped |
| increase the confidence margin from 50% to 80% (ablation) | MuRater training predictions · Training set (held-in) | accuracy: 94.3% -> 97.2% | Helped |
| train MuRater with Qwen 3-8B translations rather than GPT-4o translations | MuRater multilingual scoring · 17 evaluated languages | agreement with GPT-4o-translation MuRater: Kendall’s~Tau 0.8427–0.9044; Pearson Corr. 0.9644–0.9861 | No clear effect |
Surprising findings
- Pointwise GPT-4o scores vary notably across languages in the mid-quality range, while pairwise judgments remain consistent across languages.
- Translation quality of GPT-4o is partially but not fully responsible for multilingual performance gaps (e.g., Japanese and Thai underperform despite comparable token ratios).
Limitations
- Relies on proprietary GPT-4o for annotation and translation, introducing biases.
- English raters focus on factual/informational content, so the rater performs worse on narrative and creative domains.
- Covers only 17 target languages; translation quality varies (Japanese and Thai lower).
Experimental setup
Datasets: FineWeb-2 FineWeb-Edu classifier annotations DCLM fastText classifier annotations MuBench lm-evaluation-harness tasks CMMLU IndoMMLU JMMLU AMMLU VMLU
Benchmarks: ARC-Easy ARC-Challenge SciQ LogiQA TriviaQA BoolQ HellaSwag PIQA OpenBookQA WinoGrande Natural Questions MMLU
Metrics: accuracy average accuracy MSE slope Kendall's Tau Pearson correlation
Baselines: Uniform QuRater AskLLM DCLM FineWeb-Edu FineWeb2-HQ HPLT-2 FineWeb-2 QuRater-M
Builds on: QuRater Ask-LLM DCLM FineWeb-Edu BGE-M3
Closest papers by meaning
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.92)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.90)
- QuRating: Selecting High-Quality Data for Training Language Models (ICML 2024 · similarity 0.89)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.89)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.88)
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.