Research map / Large language models / Language, safety and interpretability
How Text Quality Interventions Reshape Neural Scaling Laws for LLMs: Empirical Study
Newsha Ardalani, Feiyang Kang, Michael Kuchnik, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li, Carole-Jean Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add exact deduplication (ablation) | QualityPajama CC data · compute scaling law | compute efficiency: 100× gain in compute efficiency with data volume reduced to 83% | Improved |
| use fuzzy dedup threshold 0.7 instead of 0.9 (ablation) | dedupe_0.7 · compute scaling law | compute required: 3× less compute than dedupe_0.9 | Improved |
| use fuzzy dedup threshold 0.7 instead of exact dedup (ablation) | dedupe_0.7 · compute scaling law | compute required: 10× less than exact deduplication, 300× less than no deduplication | Improved |
| filter strictly to high-PageRank pages (ablation) | PageRank filtering · compute scaling law | compute efficiency | No clear effect |
| include pages not found in PageRank table (no_pr) (ablation) | PageRank filtering · compute scaling law | compute efficiency | Improved |
| mix synthetic and natural data instead of either alone (ablation) | synthetic-natural mixing · compute scaling law | projected loss | Improved |
| add NSFW filtering on top of perplexity filter (ablation) | high_perplexity · scaling exponent alpha | alpha: 0.3509 -> 0.4341 | Improved |
| change from high_perplexity to nsfw filtered data (ablation) | scaling law fit · irreducible loss E | E: 0.35 -> 0.16 | Improved |
| use heuristic filters vs synthetic data for ranking consistency | scaling components · validation-set rank consistency | Spearman correlation of A: 0.45 vs 0.81 | Mixed |
| vary data quality intervention | compute-optimal design · token-to-parameter ratio | ratio spread: up to 182× for the token-to-parameter ratio | Mixed |
| relate Zipf exponent of data to model exponent | filtered datasets · scaling exponents | correlation with alpha: -0.37 with alpha, -0.005 with beta | No clear effect |
| use exact vs fuzzy dedup threshold 1.0 vs aesthetic baseline (ablation) | deduped_1.0 · irreducible loss E | E: 0.50 -> 0.05 | Improved |
Closest papers by meaning
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.90)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.90)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.90)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.90)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)
- Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
