Research map / Large language models / Language, safety and interpretability
Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds model-based (FastText and XLM-RoBERTa embedding MLP/cosine) quality filters for multilingual FineWeb-2 using structured, knowledge-rich reference data, and releases FineWeb2-HQ covering 20 languages.
Problem
Model-based data filtering for LLM pretraining has focused on English, while multilingual web data relies mainly on heuristic filters; the paper targets structured, knowledge-rich multilingual pretraining data.
Approach
Train per-language binary classifiers (FastText or an MLP on frozen XLM-RoBERTa embeddings) that treat curated multilingual instruction/knowledge datasets as positives and random FineWeb-2 documents as negatives, then keep top-scoring documents; the MLP on MKC+ with 10% retention performs best.
Key results
- MLP MKC+ achieves the best average rank (4.35) across Chinese, German, French, Arabic, and Danish, versus 8.72 for the FineWeb-2 baseline.
- Matches baseline MMLU with as little as 15% of training tokens (about 20B tokens for high-resource languages, 16.7% of total).
- On English, MLP MKC+ (top 10%) achieves average rank 1.83 vs 3.33 for FineWeb, and multilingual 595B-token training mitigates the curse of multilinguality for French.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MLP filtering trained on MKC+ rather than using FineWeb-2 baseline data | 1B Llama pretraining · Chinese, German, French, Arabic, Danish FineTasks | average rank: 8.72 -> 4.35 | Helped |
| Use MKC+ rather than MKC to train the MLP filter (ablation) | MLP multilingual filtering · Chinese, German, French, Arabic, Danish FineTasks | average rank: 6.11 -> 4.35 | Helped |
| Replace the MLP filter with FastText filtering | Multilingual FineWeb-2 filtering · Chinese, German, French, Arabic, Danish FineTasks | average rank: 4.35 vs 7.17 | Mixed |
| Use a 10% rather than 15% or 20% retention threshold with MLP MKC+ (ablation) | MLP MKC+ multilingual filtering · Chinese, German, French FineTasks | average rank: 8.85; 9.44; 11.97 | Helped |
| Use MKC+ rather than training the MLP filter on one representative dataset alone (ablation) | MLP multilingual filtering · Chinese, German, French, Arabic, Danish FineTasks (30B tokens) | average rank: 2.52 vs 2.91; 3.17; 3.57; 4.53; 5.42 | Helped |
| Train the filter on Include-Base-44 alone rather than MKC+ (ablation) | MLP multilingual filtering · Chinese, German, French, Arabic, Danish FineTasks (30B tokens) | average rank: 2.52 -> 5.42 | Hurt |
| Train the filter on OpenAssistant-2 alone rather than MKC+ (ablation) | MLP multilingual filtering · Chinese, German, French, Arabic, Danish FineTasks (30B tokens) | average rank: 2.52 -> 4.53 | Hurt |
| Add 5% unfiltered FineWeb-2 replay data to MLP MKC+ training (ablation) | MLP MKC+ filtering · Chinese, German, French FineTasks | average rank: 5.16 -> 5.09 | No clear effect |
| Add 10% unfiltered FineWeb-2 replay data to MLP MKC+ training (ablation) | MLP MKC+ filtering · Chinese, German, French FineTasks | average rank: 5.16 -> 5.40 | Hurt |
| Add unfiltered FineWeb-2 replay data to FT MKC+ training (ablation) | FT MKC+ filtering · Chinese, German, French FineTasks | average rank: 7.51 -> 7.17; 8.66 | Mixed |
| Decontaminate training documents (ablation) | MLP MKC+ English pretraining · English SmolLM tasks (119B tokens) | average rank: 1.5000 -> 2.1111 | Hurt |
| Train a multilingual model on the refined MLP MKC+ datasets instead of a monolingual model (ablation) | 1B Llama · French FineTasks | average rank: 2.0556 -> 1.8333 | Helped |
Surprising findings
- Multilingual training on filtered data turns the curse of multilinguality into a benefit for French, with the multilingual model outperforming its monolingual counterpart at equal per-language tokens.
- Models trained only on Include-Base-44 or OpenAssistant-2 perform worse than the unfiltered baseline.
Limitations
- Experiments use only 1B models with one seed per run and no error bars.
- Uses only the first 512 tokens of each document for classification.
- Classifier training datasets have no quality guarantees for languages beyond those studied.
Experimental setup
Datasets: FineWeb-2 Aya Collection Aya Dataset MMLU OpenAssistant-2 Include-Base-44 FineWeb FineWeb-Edu DCLM
Benchmarks: FineTasks SmolLM tasks MMLU CMMLU Belebele HellaSwag ARC XNLI FQuAD
Metrics: accuracy average rank
Baselines: FineWeb-2 heuristic filtering FineWeb FineWeb-Edu DCLM Cosine similarity filter
Builds on: DCLM FineWeb-2 FineWeb-Edu FastText XLM-RoBERTa
Closest papers by meaning
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.92)
- DataComp-LM: In search of the next generation of training sets for language models (NeurIPS 2024 · similarity 0.90)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.90)
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (ICLR 2025 · similarity 0.90)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.