Research map / Large language models / Retrieval and knowledge
Synthetic continued pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes synthetic continued pretraining, instantiated as EntiGraph (entity-graph-based synthetic data augmentation), to teach a pretrained LM knowledge of a small 1.3M-token corpus, showing log-linear QA gains up to 455M synthetic tokens.
Problem
Continued pretraining is data-inefficient on small, niche corpora (around 1M tokens), where each fact appears only rarely, so models fail to acquire parametric knowledge of the documents.
Approach
Use a prompted LLM to extract entities from the small corpus and then generate relation-analysis texts over sampled entity pairs and triplets, forming a knowledge-graph-like synthetic corpus whose diversity comes from entity combinations, then continue pretraining on it.
Key results
- EntiGraph CPT on 455M tokens raises closed-book QuALITY accuracy from 39.49% (Llama 3 8B Base) to 56.22%, with log-linear scaling in synthetic token count.
- Raw CPT performs worse than the base model, and Rephrase CPT scales poorly compared with EntiGraph.
- RAG with EntiGraph CPT reaches 62.60% vs 60.35% for RAG with Llama 3 8B Base; EntiGraph alone gives >80% of RAG's absolute gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue pretraining on 455M EntiGraph tokens | Llama 3 8B Base · QuALITY closed-book QA | accuracy: 39.49% -> 56.22% | Helped |
| Continue pretraining on the raw QuALITY corpus | Llama 3 8B Base · QuALITY closed-book QA | accuracy | Hurt |
| Train on repeatedly rephrased QuALITY articles instead of EntiGraph data | Llama 3 8B Base · QuALITY closed-book QA | accuracy | Mixed |
| Add RAG to EntiGraph CPT | EntiGraph CPT · QuALITY open-book RAG | accuracy: 60.35 -> 62.60 | Helped |
| Use EntiGraph CPT rather than the base model in the RAG pipeline | Llama 3 8B Base + RAG · QuALITY open-book RAG | accuracy: 60.35 -> 62.60 | Helped |
| Add RAG | Llama 3 8B Base · QuALITY QA | accuracy: 39.49% -> 60.35% | Helped |
| Instruction-tune the continually pretrained model | Llama 3 8B Base · AlpacaEval | winrate against GPT-4: 0% -> 6.25% | Helped |
Surprising findings
- Raw continued pretraining on the 1.3M-token corpus performs worse than the base Llama 3 8B model.
- The closed-book EntiGraph CPT model (56.22%) exceeds closed-book GPT-4 (51.30%).
Limitations
- Synthetic data generation relies on a strong prompted model (gpt-4-turbo), so hallucination risk and dependence on generator capability remain.
- Bootstrapping with a weaker model generating its own synthetic data is not demonstrated.
- Evaluation limited to a single small corpus (QuALITY) and single base model family (Llama 3 8B).
Experimental setup
Datasets: QuALITY RedPajama UltraChat sQuALITY
Benchmarks: QuALITY sQuALITY
Metrics: accuracy Recall@8 normalized false claims normalized salient claims
Baselines: Raw CPT Rephrase CPT Llama 3 8B Base GPT-4 GPT-3.5 QA SFT RAG with Llama 3 8B Base
Builds on: retrieval-augmented generation chain-of-thought prompting deductive closure training
Closest papers by meaning
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.88)
- From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data (ICLR 2025 · similarity 0.86)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.86)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.86)
- How to Synthesize Text Data without Model Collapse? (ICML 2025 · similarity 0.86)
- In-Context Pretraining: Language Modeling Beyond Document Boundaries (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.