Research map / Large language models / Language, safety and interpretability
Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity
Yeongbin Seo, Gayoung Kim, Jaehyung Kim, Jinyoung Yeo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| filter with prior-based method instead of PPL-based filtering | GPT-2 1.5B pretraining on Dolma · Dolma, 20 benchmarks avg | avg normalized accuracy: 8.22 -> 9.20 | Improved |
| filter with prior-based method instead of PPL-based filtering | GPT-2 137M pretraining on Dolma · Dolma, 20 benchmarks avg | avg normalized accuracy: 5.26 -> 6.65 | Improved |
| use prior-based filter instead of PPL-based for filtering cost | data filtering of 6B-token Dolma · Dolma filtering time | filtering time: 216 GPU hrs -> 0.25 hrs | Improved |
| filter with PPL-based method rather than no filter on symbolic problem solving | GPT-2 1.5B on Dolma · Dolma, symbolic problem solving | normalized accuracy: 13.22 -> 7.91 | Hurt |
| filter with prior-based method on code corpus instead of PPL-based | GPT-2 1.5B on Pile-github · Pile-github, 6 symbolic benchmarks | avg normalized accuracy: 11.21 -> 12.03 | Improved |
| filter with prior-based method on code corpus instead of PPL-based | GPT-2 137M on Pile-github · Pile-github, 6 symbolic benchmarks | avg normalized accuracy: 9.82 -> 12.19 | Improved |
| PPL-based filtering on code corpus versus no filtering at 137M | GPT-2 137M on Pile-github · Pile-github, 6 symbolic benchmarks | avg normalized accuracy: 10.15 -> 9.82 | No clear effect |
| use prior statistics from 1% subset of corpus instead of full corpus (ablation) | prior-based filter · Dolma outlier set overlap | outlier overlap with full-corpus set | No clear effect |
Closest papers by meaning
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.87)
- Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models (ICLR 2025 · similarity 0.86)
- DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.86)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.86)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.86)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
