Research map / Large language models / Retrieval and knowledge
How Do Large Language Models Acquire Factual Knowledge During Pretraining?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how OLMo LLM checkpoints acquire injected fictional factual knowledge during continued pretraining, tracking memorization, generalization, and forgetting across injection scenarios, model sizes, pretraining stages, deduplication and batch size.
Problem
How LLMs acquire factual knowledge during pretraining and how that acquisition is affected by training conditions and forgotten over time remains poorly understood.
Approach
Injects fictional knowledge passages into intermediate OLMo pretraining checkpoints and tracks step-wise log probability of probes, defining local acquisition maxima, effectivity, and retainability to show acquisition accumulates micro-updates while forgetting follows a power law in training steps.
Key results
- Pretraining on more tokens (early vs mid vs late checkpoints) shows no significant improvement in effectivity of factual knowledge acquisition, while 7B shows higher effectivity than 1B.
- Retainability decays linearly against log(t) (R^2 > 0.80 for memorization and semantic generalization), indicating a power-law forgetting relationship.
- Compositional generalization forgets more slowly (smaller decay constant) than memorization and semantic generalization; paraphrased injection forgets more slowly than duplication, and smaller batch size (128 vs 2048) accelerates forgetting and shrinks the learnability threshold.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use paraphrased rather than duplicated injected knowledge | OLMo-7B pretraining · Fictional Knowledge, memorization and generalization | retainability decay constant | Helped |
| Increase model size from 1B to 7B | OLMo pretraining · Fictional Knowledge, all injection scenarios | effectivity | Helped |
| Train on checkpoints with more pretraining tokens | OLMo pretraining · Fictional Knowledge, all model scales and injection scenarios | effectivity | No clear effect |
| Use a smaller batch size (128 rather than 2048) | OLMo-7B Mid · Fictional Knowledge, compositional generalization | retainability | Hurt |
| Inject duplicated knowledge rather than paraphrased knowledge | OLMo-7B pretraining · Fictional Knowledge, memorization and generalization | retainability | Hurt |
| Increase batch size from 128 to 2048 | OLMo-7B Mid · Fictional Knowledge, factual knowledge acquisition | robustness to forgetting | Helped |
Surprising findings
- Effectivity of factual knowledge acquisition does not improve with later pretraining checkpoints trained on more tokens.
- Duplicated injection yields higher immediate effectivity for generalization but faster forgetting, so the advantage disappears by the end of measurement.
Limitations
- Does not evaluate generated outputs, only log probabilities, so the link to generation behavior is not established.
- Does not analyze very early pretraining dynamics.
- Batch size and learning rate effects are studied at only limited values.
Experimental setup
Datasets: Dolma v1.5 ECBD
Benchmarks: Fictional Knowledge
Metrics: log probability of target span effectivity retainability decay constant
Builds on: OLMo Dolma ECBD
Closest papers by meaning
- Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition (ICLR 2025 · similarity 0.90)
- Train Once, Answer All: Many Pretraining Experiments for the Cost of One (ICLR 2026 · similarity 0.89)
- How new data permeates LLM knowledge and how to dilute it (ICLR 2025 · similarity 0.89)
- What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.88)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.88)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.