Research map / Large language models / Retrieval and knowledge
Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses controlled synthetic biography datasets to show that LLMs only extract knowledge after fine-tuning when pretraining data is knowledge-augmented (rewritten, permuted, multiplied), and that linear probing reveals why.
Problem
Whether language models answer factual questions by genuinely extracting knowledge from training text versus by memorizing similar questions seen during training, and what conditions in pretraining enable knowledge extraction after fine-tuning.
Approach
Study knowledge storage and extraction in a fully controlled setting: pretrain small GPT2/Llama models from scratch on synthetic (bioS) and LLaMA-rewritten (bioR) biographies, then test out-of-distribution QA after fine-tuning on other individuals; show that knowledge augmentation (multiple rewrites, sentence permutation, full names) during pretraining is what enables extraction, and explain it wi
Key results
- Mixed training on bioS single achieves 86.6% OOD QA accuracy, and 77.7% on bioR single.
- Pretraining on bioS with 5 diverse biography entries per person raises QA fine-tune accuracy on unseen individuals from 9.7% to 96.6%, while the single-entry setting stays near zero.
- P-probing shows that heavy augmentation (multi5+permute) makes all six attributes linearly decodable from the person's name tokens, while the un-augmented setting only encodes them late in the biography.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mix BIO data with QA training data before fine-tuning | GPT2 · bioS single, OOD QA | accuracy | Helped |
| Mix BIO data with QA training data before fine-tuning | GPT2 · bioR single, OOD QA | accuracy | Helped |
| Add knowledge augmentation during BIO pretraining | GPT2 BIO pretraining + QA fine-tuning · bioS, OOD QA | accuracy: 9.7% to 96.6% | Helped |
| Add five biography entries per person without permutation (ablation) | GPT2 BIO pretraining + QA fine-tuning · bioS, OOD QA | accuracy: 9.7% to 41% | Helped |
| Permute the six sentences five times instead of once (ablation) | GPT2 BIO pretraining + QA fine-tuning · bioS, OOD QA | accuracy: 4.4% to 70% | Helped |
| Add one sentence permutation to single-entry biographies without multiplicity (ablation) | GPT2 BIO pretraining + QA fine-tuning · bioS, OOD QA | accuracy | Hurt |
| Replace pronouns or partial names with the person's full name throughout biographies (ablation) | GPT2 BIO pretraining + QA fine-tuning · bioS and bioR, OOD QA | accuracy | Helped |
| Increase multiplicity or the number of permutations (ablation) | GPT2 BIO pretraining + QA fine-tuning · bioS, OOD QA | accuracy | Helped |
| Increase the QA-data ratio during mixed training (ablation) | GPT2 mixed training · OOD QA | accuracy | Helped |
| Use an encoder-only BERT-like model instead of an autoregressive decoder | Knowledge extraction after pretraining and fine-tuning · BIO biography QA | accuracy | Hurt |
Surprising findings
- Pretraining on biographies without augmentation yields near-zero QA accuracy on unseen individuals regardless of model size, pretraining duration, or fine-tuning method (full or LoRA), despite 99%+ memorization accuracy.
- Mixed training lets the model first use QA data to encode knowledge and then align it with biographies, a 'study to pass the test' pattern.
Limitations
- Synthetic bioS data with uniform, independent attributes limits transfer to real internet pretraining.
- Experiments use small GPT2/Llama models trained from scratch (124M-682M), not frontier-scale LLMs.
- Exact-match QA accuracy disregards partial matches or synonyms.
Experimental setup
Datasets: bioS (synthetic biographies) bioR (LLaMA-generated biographies)
Benchmarks: bioS QA (out-of-distribution individuals) bioR QA (out-of-distribution individuals)
Metrics: exact-match QA accuracy first-token accuracy P-probing accuracy
Baselines: BERT GBERT
Builds on: LoRA GPT-2 Llama
Closest papers by meaning
- KScope: A Framework for Characterizing the Knowledge Status of Language Models (NeurIPS 2025 · similarity 0.89)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.89)
- Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.89)
- How new data permeates LLM knowledge and how to dilute it (ICLR 2025 · similarity 0.89)
- Delving into the Reversal Curse: How Far Can Large Language Models Generalize? (NeurIPS 2024 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.