AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Synthetic Bootstrapped Pretraining

Zitong Yang, Aonan Zhang, Hong Liu, Tatsunori Hashimoto, Emmanuel Candes, Chong Wang, Ruoming Pang

ICLR 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add synthetic documents from the SBP synthesizer to pretraining data3B model, 200B-scale · Average QA accuracyaccuracy: 47.66 -> +2.32 (oracle +5.54)Improved
add synthetic documents from the SBP synthesizer3B model, 1T-scale · Average QA accuracyaccuracy: 55.35 -> +0.84 (oracle +1.73)Improved
add synthetic documents from the SBP synthesizer6B model, 1T-scale · Average QA accuracyaccuracy: 58.29 -> +1.32 (oracle +2.26)Improved
add SBP synthetic data3B model, 1T-scale · OpenWebText2 and LAMBADA perplexityperplexity: -0.02 and -0.03No clear effect
increase synthetic tokens from 0B to 100B (ablation)SBP, 200B-scale · OpenWebText2, LAMBADA perplexity and QA benchmarksperplexity and accuracy: perplexity decreases monotonically; QA peaks at 75BMixed
use 125B synthetic tokens rather than 0B or 250B (ablation)SBP, 3B at 1T-scale · All benchmarksbenchmark performance: 125B best across all benchmarks except LAMBADA perplexityImproved
increase synthetic share with larger model (6B uses 250B vs 125B for 3B) (ablation)SBP, 6B at 1T-scale · Benchmarksbenchmark performance: optimal about 250B vs 125B for 3BImproved
train the synthesizer on random document pairs instead of nearest-neighbor pairs (ablation)SBP synthesizer · Embedding similarity to seedmean similarity: 0.66 -> 0.32Hurt
use constant small synthesizer-tuning learning rate (5e-6) instead of cosine decay with larger LR (ablation)SBP synthesizer-tuning · Synthesized text qualitytext qualityImproved
scale synthesizer to 1T-scale and larger modelSBP synthesizer · Non-factual ratenon-factual rate: 15.1% -> 8.7% -> 6.5%Improved
scale synthesizer data (12M to 60M documents)SBP synthesizer · Pair-irrelevancepair-irrelevance: 25.6% -> 7.8%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy