Research map / Large language models / Language, safety and interpretability
Synthetic Bootstrapped Pretraining
Zitong Yang, Aonan Zhang, Hong Liu, Tatsunori Hashimoto, Emmanuel Candes, Chong Wang, Ruoming Pang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add synthetic documents from the SBP synthesizer to pretraining data | 3B model, 200B-scale · Average QA accuracy | accuracy: 47.66 -> +2.32 (oracle +5.54) | Improved |
| add synthetic documents from the SBP synthesizer | 3B model, 1T-scale · Average QA accuracy | accuracy: 55.35 -> +0.84 (oracle +1.73) | Improved |
| add synthetic documents from the SBP synthesizer | 6B model, 1T-scale · Average QA accuracy | accuracy: 58.29 -> +1.32 (oracle +2.26) | Improved |
| add SBP synthetic data | 3B model, 1T-scale · OpenWebText2 and LAMBADA perplexity | perplexity: -0.02 and -0.03 | No clear effect |
| increase synthetic tokens from 0B to 100B (ablation) | SBP, 200B-scale · OpenWebText2, LAMBADA perplexity and QA benchmarks | perplexity and accuracy: perplexity decreases monotonically; QA peaks at 75B | Mixed |
| use 125B synthetic tokens rather than 0B or 250B (ablation) | SBP, 3B at 1T-scale · All benchmarks | benchmark performance: 125B best across all benchmarks except LAMBADA perplexity | Improved |
| increase synthetic share with larger model (6B uses 250B vs 125B for 3B) (ablation) | SBP, 6B at 1T-scale · Benchmarks | benchmark performance: optimal about 250B vs 125B for 3B | Improved |
| train the synthesizer on random document pairs instead of nearest-neighbor pairs (ablation) | SBP synthesizer · Embedding similarity to seed | mean similarity: 0.66 -> 0.32 | Hurt |
| use constant small synthesizer-tuning learning rate (5e-6) instead of cosine decay with larger LR (ablation) | SBP synthesizer-tuning · Synthesized text quality | text quality | Improved |
| scale synthesizer to 1T-scale and larger model | SBP synthesizer · Non-factual rate | non-factual rate: 15.1% -> 8.7% -> 6.5% | Improved |
| scale synthesizer data (12M to 60M documents) | SBP synthesizer · Pair-irrelevance | pair-irrelevance: 25.6% -> 7.8% | Improved |
Closest papers by meaning
- Synthetic continued pretraining (ICLR 2025 · similarity 0.88)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.87)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.87)
- Nemotron-CLIMB: Clustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (NeurIPS 2025 · similarity 0.86)
- In-Context Pretraining: Language Modeling Beyond Document Boundaries (ICLR 2024 · similarity 0.86)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
