Research map / Large language models / Language, safety and interpretability
RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
Zichun Yu, Chenyan Xiong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the rephraser with RL using quality and faithfulness rewards instead of using organic data only | web recycling · DCLM 400M/1.4B/2.8B | Core score: 3.7%-14.5% relative improvements | Improved |
| add RL (RePro) in place of direct prompting of the base rephraser (ablation) | RePro · 400M setting | Core score: 0.19847 -> 0.21658 | Improved |
| add RL in place of SFT on GPT-4o rephrasings (ablation) | RePro · 400M setting | Core score: 0.19216 -> 0.21658 | Improved |
| add faithfulness rewards to RL (ablation) | RePro · 400M setting | Core score: 0.19456 -> 0.21658 | Improved |
| add SFT warm-up before RL (ablation) | RePro · 400M setting | Core score: 0.21658 -> 0.20816 | Hurt |
| add the reward descriptions to the prompting baseline (Prompting+) (ablation) | direct prompting · 400M setting | Core score: 0.19847 -> 0.19910 | No clear effect |
| also recycle high-quality organic data rather than only the non-high-quality part (ablation) | RePro · 400M setting | Core score: 0.21050 -> 0.21658 | Improved |
| merge organic high-quality data with recycled data instead of recycled alone (ablation) | RePro · 400M setting | Core score: 0.20231 -> 0.21749 | Improved |
| use RePro recycled data instead of organic high-quality data alone | 400M model · 400M setting | Core score: +6.5% on Core | Improved |
| use RePro instead of ReWire on the same organic pool | web recycling · 400M setting | Core score: by 9.0% on Core | Improved |
| spend compute on rephrasing with RePro instead of longer training on organic data | 1.4B model · 1B setting, equalized compute | Core score: 0.27871 -> 0.29746 | Improved |
| increase the unique token budget B from 21.6B to 28.8B | RePro · 1B setting | performance | Hurt |
Closest papers by meaning
- Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels (ICLR 2026 · similarity 0.88)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.86)
- Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale (ICML 2025 · similarity 0.86)
- RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting (AAAI 2024 · similarity 0.86)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.86)
- DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
