Research map / Large language models / Language, safety and interpretability
Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProX, which trains small (0.3B–1.7B) refining models via SFT on LLM-annotated data to generate executable per-document and per-chunk cleaning programs, improving pre-training corpora quality and continual pre-training efficiency.
Problem
Pre-training corpus quality is improved mainly by fixed human-crafted heuristic rules, which cannot adapt to individual examples, while LLM-based synthesis is costly and hallucination-prone.
Approach
Treat data refinement as a programming task: a small language model, fine-tuned on LLM-annotated doc-program pairs, generates executable function calls (drop_doc, remove_lines, normalize, keep_chunk) for each document and chunk, which a Python executor applies to produce refined corpora.
Key results
- ProX-D+C on RedPajama-V2 improves 10-task zero-shot average from 42.1 (raw) to 44.6 with 0.7B model, beating best rule-based filter (42.8) and MATES by about 2 points.
- On OpenWebMath continual pre-training, ProX improves CodeLlama-7B by 20.3% average (CPT baseline +14.1%), matching Llemma-7B trained on 200B tokens with 10B tokens.
- ProX-refined data gives +2.0% on RedPajama-V2, +3.1% on C4, +2.4% on FineWeb, +0.9% on FineWeb-Edu, +1.7% on DCLM at 50B tokens scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add chunk-level programming after document-level programming (ablation) | ProX-D on RedPajama-V2 · RedPajama-V2, 10-task zero-shot | average accuracy: 43.5 -> 44.6 | Helped |
| Refine RedPajama-V2 with ProX-D+C instead of using raw data | TLM-s pre-training · RedPajama-V2, 10-task zero-shot | average accuracy: 42.1 -> 44.6 | Helped |
| Use ProX-D+C instead of FineWeb rule-based filtering | TLM-s pre-training · RedPajama-V2, 10-task zero-shot | average accuracy: 42.8 -> 44.6 | Helped |
| Refine OpenWebMath with ProX instead of using original data | CodeLlama-7B continual pre-training · OpenWebMath | average accuracy: 43.2 -> 49.4 | Helped |
| Refine OpenWebMath with ProX instead of using original data | Llama-2-7B continual pre-training · OpenWebMath | average accuracy: 42.8 -> 46.1 | Helped |
| Refine OpenWebMath with ProX instead of using original data | Mistral-7B continual pre-training · OpenWebMath | average accuracy: 54.8 -> 59.2 | Helped |
| Refine OpenWebMath with ProX instead of using original data | lm continual pre-training · OpenWebMath | average accuracy: 21.5 -> 25.7 | Helped |
| Apply rule-based filtering to OpenWebMath instead of continual pre-training on the original corpus | lm · OpenWebMath | average accuracy: 21.5 -> 18.4 | Hurt |
| Use FineWeb-ProX curation instead of FineWeb-Edu selection | TLM-s pre-training · FineWeb, 10-task zero-shot | average accuracy: 45.2 -> 45.2 | No clear effect |
Surprising findings
- Rule-based filtering on OpenWebMath degrades continual pre-training by 3.1% relative to training on original data.
- The 0.35B refining model keeps only 23.2% of tokens versus 28.8% for larger refiners, yet the smallest refiner still gives comparable downstream gains.
Limitations
- Large-scale inference for refining still requires substantial compute overhead.
- Normalization sub-task is hard to evaluate, so chunk-level evaluation uses only line-removal F1.
- Operations are limited to deletion and replacement; reformatting and rephrasing are not included.
Experimental setup
Datasets: RedPajama-V2 C4 FineWeb FineWeb-Edu DCLM-baseline-1.0 OpenWebMath
Benchmarks: ARC-C ARC-E CSQA HellaSwag MMLU OBQA PIQA SIQA WinoGrande SciQ LogiQA GSM8K
Metrics: zero-shot normalized accuracy few-shot accuracy F1 average accuracy
Baselines: Gopher rules C4 rules FineWeb rules DSIR DsDm MATES QuRating TinyLlama-1.1B-3T SheadLlama InstructionLM-1.3B cosmo-1.8B Rho-1
Builds on: FineWeb-Edu MATES Rho-1 llama-factory vLLM
Closest papers by meaning
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.89)
- DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.88)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.88)
- RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.88)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.88)
- MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.