Research map / Large language models / Language, safety and interpretability

A Bitter Lesson for Data Filtering

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Scaling studies on Common Crawl subsets show that with enough model size and training steps, unfiltered data matches or beats standard filters (DCLM-Baseline, RefinedWeb) and tolerates injected shuffled or random junk data.

Problem

Whether data filtering for LLM pretraining remains necessary in the high-compute, data-scarce regime, where models are trained for many epochs on small pools.

Approach

Measure the best achievable loss over model size and training steps (unconstrained by compute) rather than at fixed budget, and find that large models trained long enough benefit from unfiltered Common Crawl and even from injected shuffled-word or random-string data; extrapolate a compute threshold (~1e30 FLOPs) where the full 240T-token pool beats RefinedWeb.

Key results

• Unfiltered 670M CC pool beats all five filtered variants once models reach roughly 330M+ parameters and enough steps; the pool becomes the Pareto-optimal choice at high compute.
• On the 330M model, shuffled-word injections of +200% and smaller surpass the clean pool loss after about 11B training tokens.
• Scaling laws extrapolate that the full 240T-token DCLM-Pool CC becomes better than RefinedWeb at about 1e+30 FLOPs (R^2 above 0.99).

Surprising findings

• Large models benefit from appended shuffled-word documents and show little harm from random-string junk data.
• Filtering can be Pareto-dominated: at every compute level some looser filters beat the repetition-filtered dataset.

Limitations

• Limited to dense transformers with no curricula, data weights, or post-training; MoE behavior untested.
• Extrapolated 1e30 FLOP threshold is far beyond experiments and relies on fitted scaling laws.
• Factuality analysis is a brief keyword-matched study on a few MMLU subjects; rare harmful content may be missed.

Experimental setup

Datasets: DCLM-Pool DCLM-Baseline RefinedWeb C4 FineWeb-Edu Cosmopedia MMLU

Benchmarks: C4 FineWeb-Edu Cosmopedia PIQA ARC-Easy SocialIQA

Metrics: negative log-likelihood accuracy compute-optimal Pareto frontier crossing point (epochs)

Baselines: DCLM-Baseline RefinedWeb English filter Repetition filter Stop word filter

Builds on: Chinchilla scaling laws DCLM Meta Lingua Muennighoff data-constrained scaling

Closest papers by meaning

• Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic (CVPR 2024 · similarity 0.89)
• Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality (NeurIPS 2025 · similarity 0.89)
• Data Filtering Networks (ICLR 2024 · similarity 0.88)
• Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.87)
• Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.86)
• An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.86)