AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

Zichun Yu, Chenyan Xiong

ICML 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train the rephraser with RL using quality and faithfulness rewards instead of using organic data onlyweb recycling · DCLM 400M/1.4B/2.8BCore score: 3.7%-14.5% relative improvementsImproved
add RL (RePro) in place of direct prompting of the base rephraser (ablation)RePro · 400M settingCore score: 0.19847 -> 0.21658Improved
add RL in place of SFT on GPT-4o rephrasings (ablation)RePro · 400M settingCore score: 0.19216 -> 0.21658Improved
add faithfulness rewards to RL (ablation)RePro · 400M settingCore score: 0.19456 -> 0.21658Improved
add SFT warm-up before RL (ablation)RePro · 400M settingCore score: 0.21658 -> 0.20816Hurt
add the reward descriptions to the prompting baseline (Prompting+) (ablation)direct prompting · 400M settingCore score: 0.19847 -> 0.19910No clear effect
also recycle high-quality organic data rather than only the non-high-quality part (ablation)RePro · 400M settingCore score: 0.21050 -> 0.21658Improved
merge organic high-quality data with recycled data instead of recycled alone (ablation)RePro · 400M settingCore score: 0.20231 -> 0.21749Improved
use RePro recycled data instead of organic high-quality data alone400M model · 400M settingCore score: +6.5% on CoreImproved
use RePro instead of ReWire on the same organic poolweb recycling · 400M settingCore score: by 9.0% on CoreImproved
spend compute on rephrasing with RePro instead of longer training on organic data1.4B model · 1B setting, equalized computeCore score: 0.27871 -> 0.29746Improved
increase the unique token budget B from 21.6B to 28.8BRePro · 1B settingperformanceHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy