Research map / Large language models / Retrieval and knowledge
Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses EK-FAC influence functions over 5M pretraining documents (~2.5B tokens) to show that LLM reasoning on simple math tasks relies on procedural knowledge from documents rather than retrieval of answers, unlike factual recall.
Problem
Benchmark results cannot separate generalisation from contamination because pretraining corpora are too large for train-test separation; it is unclear what pretraining data LLMs rely on when reasoning versus recalling facts.
Approach
Scale EK-FAC influence functions to 7B and 35B transformers to rank pretraining documents by influence on reasoning and factual completions, then compare correlations across queries, magnitude of influence, answer presence, and source datasets.
Key results
- Influence scores of documents correlate significantly across queries of the same reasoning task, but not for factual queries or across different reasoning tasks.
- The answer to factual queries appears in the top 500 documents for 55% of queries for the 7B model versus 7.4% for reasoning queries; for the 35B model, 30% for factual and 0% for reasoning.
- Top influential documents for slope queries often contain procedures for computing slope in code or math, e.g., a JavaScript eqOfLine function.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EK-FAC influence scores rather than TracIn to select positively influential training examples for removal | GPT-2-small fine-tuned on Wikitext-2 · Wikitext-2 | perplexity: 22.45 ± 0.02 -> 23.05 ± 0.05 (k=300) | Hurt |
| Use EK-FAC influence scores rather than random selection to select positively influential training examples for removal | GPT-2-small fine-tuned on Wikitext-2 · Wikitext-2 | perplexity: 22.15 ± 0.05 -> 23.05 ± 0.05 (k=300) | Hurt |
| Remove the most negatively influential training examples instead of randomly selected examples | GPT-2-small fine-tuned on Wikitext-2 · Wikitext-2 | perplexity: 24.85 ± 0.10 -> 23.32 ± 0.04 (k=300) | Helped |
| Use EK-FAC influence scores rather than random selection to select fine-tuning examples for removal | Command 2 7B fine-tuned on DROP · DROP | accuracy: 0.57 ± 0.06 -> 0.38 ± 0.05 (k=2000) | Hurt |
| Use EK-FAC influence scores rather than TracIn to select fine-tuning examples for removal | Command 2 7B fine-tuned on DROP · DROP | accuracy: 0.43 ± 0.06 -> 0.38 ± 0.05 (k=2000) | Hurt |
| Use EK-FAC influence scores rather than random selection to select fine-tuning examples for removal | Command 2 7B fine-tuned on RACE · RACE | accuracy: 0.81 ± 0.04 -> 0.74 ± 0.05 (k=2500) | Hurt |
| Use EK-FAC influence scores rather than TracIn to select fine-tuning examples for removal | Command 2 7B fine-tuned on RACE · RACE | accuracy: 0.79 ± 0.04 -> 0.74 ± 0.05 (k=2500) | Hurt |
| Use approximate SVD instead of full SVD for query gradients (ablation) | EK-FAC influence scoring · Wikitext-2 influence-score comparison | Pearson R: 0.96 ± 0.01 -> 0.96 ± 0.01 | No clear effect |
| Approximate EK-FAC matrices using a 2-block diagonal approximation (ablation) | EK-FAC influence scoring with approximate SVD · Wikitext-2 influence-score comparison | Pearson R: 0.96 ± 0.01 -> 0.95 ± 0.00 | Hurt |
| Use a 4-block rather than 2-block diagonal EK-FAC approximation (ablation) | EK-FAC influence scoring with approximate SVD · Wikitext-2 influence-score comparison | Pearson R: 0.95 ± 0.00 -> 0.93 ± 0.00 | Hurt |
| Compare reasoning queries with factual queries for the frequency of answers among top-ranked documents | Command R 7B · Top 500 influential documents | queries with answer found: 55% factual vs 7.4% reasoning | Mixed |
| Compare reasoning queries with factual queries for the frequency of answers among top-ranked documents | Command R 35B · Top 500 influential documents | queries with answer found: 30% factual; never for reasoning | Mixed |
Surprising findings
- Answers to reasoning steps are often present in the 2.5B token sample yet not among the highly influential documents.
- For the factual questions the top-influence documents show a less steep power law than reasoning, the opposite of the expected direction.
Limitations
- Influence computed on only a 2.5B token sample of pretraining data, not the full training set, so absence of answers may reflect sampling.
- Only feed-forward (MLP) parameters are considered; attention is treated as fixed, and many approximations (EK-FAC, block-diagonal, SVD) are used.
- Small query sets (40 reasoning, 40 factual) limited to simple mathematical tasks; proprietary models and data.
Experimental setup
Datasets: Cohere pretraining data sample (5M documents) Wikitext-2 DROP RACE
Benchmarks: Two-step arithmetic queries Slope queries Linear equation queries Factual queries Control queries DROP RACE Wikitext-2
Metrics: Pearson correlation of influence scores influence per nat of query information answer presence in top 500 documents counterfactual retraining accuracy/perplexity
Baselines: TracIn Random removal Command R+
Builds on: EK-FAC influence functions Grosse et al. 2023 influence functions at scale TracIn kronfluence
Closest papers by meaning
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.91)
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.90)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.90)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.90)
- Position: General Intelligence Requires Reward-based Pretraining (ICML 2025 · similarity 0.90)
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.