Research map / Large language models / Language, safety and interpretability
Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Estimates the knowledge storage capacity of language models in bits using synthetic biography datasets, finding GPT2-style models store about 2 bits per parameter, even when quantized to int8.
Problem
Quantifying how much factual knowledge (in bits) a language model can store as a function of its size when adequately trained, and how architecture, training duration, quantization, MoE sparsity, and data quality affect this capacity.
Approach
Define a controlled synthetic knowledge-tuple dataset (bioS/bioD) with a derived bit-complexity lower bound, so that model capacity can be measured as knowledge bits per parameter (capacity ratio) rather than via loss or benchmarks; find a consistent ~2 bits/param peak ratio for GPT2 trained with 1000 exposures.
Key results
- GPT2 models consistently reach a peak capacity ratio of about 2 bits per parameter after 1000 exposures, with R(F) at most 2.3 across all models.
- Quantizing to int8 does not reduce capacity, while int4 reduces it to about 0.7 bits per parameter.
- LLaMA/Mistral architectures show about 1.3x lower capacity than GPT2 at 100 exposures; junk data reduces useful-knowledge capacity by up to 20x, improved to 2x by prepending domain names.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train for 1000 exposures rather than 100 exposures | GPT2 · bioS(N) | capacity ratio (bits/parameter): 1bit/param -> 2bit/param | Helped |
| Use LLaMA/Mistral architecture rather than GPT2 | GPT2 · bioS(N), 100 exposures | capacity ratio: 1.3x lower than GPT2's | Hurt |
| Replace GPT2 with LLaMA/Mistral architecture | GPT2 · bioS(N), 1000 exposures | capacity ratio | No clear effect |
| Quantize GPT2 to int8 (ablation) | GPT2 · bioS(N), 1000 exposures | capacity ratio (bits/parameter): 2bit/param | No clear effect |
| Quantize GPT2 to int4 (ablation) | GPT2 · bioS(N), 1000 exposures | capacity ratio (bits/parameter): 2bit/param -> 0.7bit/param | Hurt |
| Use MoE with 32 experts (ablation) | GPT2 base scaling laws · bioS(N) | capacity ratio: 1.3x | Hurt |
| Add junk data at a 1:7 useful-to-junk token ratio (ablation) | GPT2 · bioS(N), 100 useful-knowledge exposures | useful-knowledge capacity: 20x | Hurt |
| Prepend a special token to useful knowledge (ablation) | GPT2 with junk data · bioS(N), useful-to-junk tokens 1:7, 100 useful-knowledge exposures | useful-knowledge capacity loss factor: 20x -> 2x | Helped |
| Reduce GPT2 MLP size by 1/4 (ablation) | GPT2 · bioS(N), 1000 exposures | capacity ratio | No clear effect |
| Remove all GPT2 MLP layers (ablation) | GPT2 · bioS(N), 1000 exposures | capacity ratio | No clear effect |
| Tie the embedding and output-layer weights (ablation) | LLaMA · bioS(N), 1000 exposures, tiny models (<10M) | capacity ratio | Helped |
| Rewrite the same knowledge using diverse sentence templates (ablation) | GPT2 · bioS(N) versus bioS-single(N), 1000 exposures | capacity ratio | Helped |
Surprising findings
- Removing MLP layers entirely does not change the 2-bit/param capacity, suggesting attention layers also store knowledge.
- Rewriting the same data multiple times (diversity) does not hurt and may even increase capacity.
Limitations
- Results are on synthetic biography-style data with a fixed knowledge base, so generalization to real pretraining corpora is unverified.
- Only the peak capacity ratio is reported and the 1000-exposure setting is the main focus; quantization methods beyond GPTQ were not explored.
Experimental setup
Datasets: bioS bioD bioR argilla-style bioS variant
Benchmarks: bioS synthetic biography knowledge accuracy extractable knowledge accuracy after LoRA QA fine-tuning
Metrics: capacity ratio (bits per parameter) knowledge accuracy cross-entropy loss
Baselines: LLaMA Mistral
Builds on: Physics of Language Models Part 3.1 GPTQ LoRA rotary positional embedding
Closest papers by meaning
- The Rise of Parameter Specialization for Knowledge Storage in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.88)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.88)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.88)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.