Research map / Large language models / Retrieval and knowledge
Physics of Language Models: Part 3.2, Knowledge Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controlled synthetic-biography experiments show GPT2/Llama/Mistral models store and retrieve knowledge well but fail at knowledge classification, comparison, and inverse search without chain-of-thought hints.
Problem
Whether language models can flexibly manipulate factual knowledge stored in their weights (retrieval, classification, comparison, inverse search) for downstream QA after pretraining and instruction-style finetuning, without data contamination.
Approach
Uses fully controlled synthetic biography datasets (bioS/bioR) with held-out individuals, so knowledge is known to be stored; then finetunes on manipulation QAs and tests out-of-distribution generalization, with and without CoT-style hints during training and testing.
Key results
- Models finetuned on QA reach near-perfect partial/dual retrieval when augmented data is used, but birth-year retrieval can fail while birth-date retrieval succeeds.
- Parity-of-birth-month classification needs about 10,000 training samples to reach 75% accuracy without hints, despite O(12) theoretical sample complexity.
- Inverse search (name from attributes) is near 0% for models trained without reversed data, regardless of model or data size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoT hints during both training and inference | Knowledge classification and comparison models · Synthetic knowledge classification and comparison | accuracy: 91.0% hint accuracy; 94.2% test accuracy (with hint) | Helped |
| Include CoT hint examples during training but remove hints at test time (ablation) | Knowledge classification and comparison models · Synthetic knowledge classification and comparison | test accuracy | No clear effect |
| Add CoT hints at both training and testing (ablation) | Knowledge classification and comparison models · Synthetic knowledge classification and comparison | accuracy: 78.1% hint accuracy; 61.5% test accuracy (with hint) | Helped |
| Fine-tune for single-knowledge extraction before manipulation (ablation) | BIO-pretrained model · Synthetic knowledge classification and comparison | manipulation accuracy | No clear effect |
| Train without CoT examples (ablation) | Knowledge classification and comparison models · Synthetic knowledge classification and comparison | test accuracy: 75% accuracy with 10,000 training samples | Hurt |
| Train without hints for major ranking (ablation) | Knowledge comparison model · Major ranking | test accuracy: 85% test accuracy with 50,000 training samples | Hurt |
| Train without hints for ranking 100 majors (ablation) | Knowledge comparison model · 100-major ranking | accuracy: 1% with 2.5 million training samples | Hurt |
| Compare celebrity birth dates over 1900–1910 instead of 1900–1950 | GPT-4 · Celebrity birth-date comparison | accuracy: 71.1% -> 52.3% | Hurt |
| Pretrain with five biography entries and shuffled sentences | Knowledge retrieval model · Dual knowledge retrieval | accuracy | Helped |
| Retrieve the company name before the company city (ablation) | Knowledge retrieval model pretrained on multi5+fullname · Dual knowledge retrieval | accuracy | Helped |
| Retrieve the company city before the company name (ablation) | Knowledge retrieval model pretrained on multi5+fullname · Dual knowledge retrieval | accuracy | Hurt |
| Retrieve only a later token, such as the birth year, rather than the full birth date (ablation) | Knowledge retrieval model · Partial birth-date retrieval | accuracy | Hurt |
Surprising findings
- Ranking 100 majors barely beats random guessing even with 2.5 million training samples, without hints.
- GPT-4 accuracy comparing birth dates drops from 71.1% (1900-1950 celebrities) to 52.3% (1900-1910), suggesting dependence on training frequency.
Limitations
- Results rely on synthetic, template-generated biographies with controlled attributes; generalization to real pretraining data is argued rather than fully shown.
- GPT-4 experiments are illustrative only because its training data is uncontrolled.
Experimental setup
Datasets: bioS synthetic biographies bioR close-to-real biographies generated by LLaMA
Benchmarks: bioS synthetic QA tasks (retrieval, classification, comparison, inverse search) WikiBio-based GPT-4 tests
Metrics: out-of-distribution test accuracy random-guess baseline comparison
Baselines: GPT-4 GPT-3.5 Llama-3
Builds on: Physics of Language Models Part 3.1 (knowledge storage and extraction) LoRA GPT-2 Llama Mistral
Closest papers by meaning
- The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More (NeurIPS 2024 · similarity 0.88)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.88)
- Enhancing Elusive Clues in Knowledge Learning by Contrasting Attention of Language Models (AAAI 2025 · similarity 0.87)
- Delving into the Reversal Curse: How Far Can Large Language Models Generalize? (NeurIPS 2024 · similarity 0.87)
- Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons (AAAI 2024 · similarity 0.87)
- From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.