AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining

Thiziri Nait Saada, Louis Béthune, Michal Klein, David Grangier, marco cuturi, Pierre Ablin

ICML 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
select more aggressively with CQF (smaller k) (ablation)350M model trained on RedPajama-V2 · ARC-Easydownstream accuracyImproved
use openwebmath as HQ set with very small k (ablation)CQF 350M model · ARC-Easydownstream accuracyHurt
select more aggressively with CQF (smaller k) (ablation)350M model, HQ sets other than ARC-Easy · HQ set next-token lossloss on HQ setHurt
use ARC-Easy as HQ set with smaller k (ablation)CQF 350M model · ARC-Easydownstream accuracy and HQ lossImproved
apply CQF again on a CQF-expanded HQ set instead of training directly on that HQ setOpenOrca-based HQ set · ARC-Easydownstream accuracy (%): 50.1% -> 53.8%Improved
use CRISP importance sampling instead of CQF when HQ set is a curated dataset (OpenOrca) (ablation)350M model · ARC-Easy downstream accuracydownstream accuracyHurt
use CRISP importance sampling instead of CQF when HQ set is ARC-Easy itself (ablation)350M model · ARC-Easydownstream accuracyImproved
fine-tune on top 10% CQF-scored decile of HQ set instead of the full HQ set (same tokens) (ablation)1.3B model, KnowledgePile and OpenOrca · ARC-Easy, MMLU, ARC-Challenge, Reward Benchaccuracy gain vs baseImproved
train on CQF-selected subsets as a quality axis (data-conditioning test) (ablation)OpenOrca CQF(k) · data-conditioning matrixL(k,k') - L(k',k')No clear effect
train on better token-permutation quality (Perm) datasets (ablation)RedPajama-V2 Perm(k) · data-conditioning matrixL(k,k') - L(k',k')Improved
scale model size (125M to 1.3B) and tokens for data-conditioning (ablation)Perm quality axis · scaling-law fitdata-conditioning regionsNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy