Research map / Large language models / Language, safety and interpretability
Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
Thiziri Nait Saada, Louis Béthune, Michal Klein, David Grangier, marco cuturi, Pierre Ablin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select more aggressively with CQF (smaller k) (ablation) | 350M model trained on RedPajama-V2 · ARC-Easy | downstream accuracy | Improved |
| use openwebmath as HQ set with very small k (ablation) | CQF 350M model · ARC-Easy | downstream accuracy | Hurt |
| select more aggressively with CQF (smaller k) (ablation) | 350M model, HQ sets other than ARC-Easy · HQ set next-token loss | loss on HQ set | Hurt |
| use ARC-Easy as HQ set with smaller k (ablation) | CQF 350M model · ARC-Easy | downstream accuracy and HQ loss | Improved |
| apply CQF again on a CQF-expanded HQ set instead of training directly on that HQ set | OpenOrca-based HQ set · ARC-Easy | downstream accuracy (%): 50.1% -> 53.8% | Improved |
| use CRISP importance sampling instead of CQF when HQ set is a curated dataset (OpenOrca) (ablation) | 350M model · ARC-Easy downstream accuracy | downstream accuracy | Hurt |
| use CRISP importance sampling instead of CQF when HQ set is ARC-Easy itself (ablation) | 350M model · ARC-Easy | downstream accuracy | Improved |
| fine-tune on top 10% CQF-scored decile of HQ set instead of the full HQ set (same tokens) (ablation) | 1.3B model, KnowledgePile and OpenOrca · ARC-Easy, MMLU, ARC-Challenge, Reward Bench | accuracy gain vs base | Improved |
| train on CQF-selected subsets as a quality axis (data-conditioning test) (ablation) | OpenOrca CQF(k) · data-conditioning matrix | L(k,k') - L(k',k') | No clear effect |
| train on better token-permutation quality (Perm) datasets (ablation) | RedPajama-V2 Perm(k) · data-conditioning matrix | L(k,k') - L(k',k') | Improved |
| scale model size (125M to 1.3B) and tokens for data-conditioning (ablation) | Perm quality axis · scaling-law fit | data-conditioning regions | No clear effect |
Closest papers by meaning
- QuRating: Selecting High-Quality Data for Training Language Models (ICML 2024 · similarity 0.89)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.89)
- The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph (ICML 2025 · similarity 0.88)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.88)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.87)
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
