Research map / Large language models / Language, safety and interpretability

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts

ICML 2025 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the extractive structures framework (informative, upstream, downstream components) to explain how LMs generalize to implications of finetuned facts, and tests a pretraining mechanism via data-ordering and weight-grafting experiments on OLMo-7b and other 7-9B models.

Problem

Pretrained LMs can generalize to implications of facts they are finetuned on (out-of-context reasoning), but the mechanisms enabling this and how they are learned during pretraining are poorly understood.

Approach

Frames out-of-context reasoning as a communication problem and defines causal, linearized component scores (informative, upstream, downstream) to localize components that store finetuned facts as weight changes and that query/process them; hypothesizes extractive structures form during pretraining when implications of already-known facts are encountered, predicting a data-ordering effect and a wei

Key results

• Freezing early-middle layers (first 24) during finetuning raised First-hop implication mean rank from 0.00 to 6.50 while Second-hop stayed at 0.50; freezing late layers (last 8) raised Second-hop implication rank to 6.60 with First-hop at 0.10.
• Weight grafting lowered counterfactual implication mean rank from 9.13 to 1.13, versus 8.38 for the control graft.
• Data ordering: Impl-first continued pretraining yielded worse OCR than Facts-first or Joint orderings on OLMo-7b, with the effect present but weaker and more hyperparameter-sensitive in Llama 3-8b, Gemma 2-9b, and Qwen 2-7b.

Measured effects reported in the paper

Surprising findings

• Fact learning occurs at both early and late layers, and these enable different forms of generalization (first-hop vs second-hop), which poses obstacles for localization-based knowledge editing.
• Attention heads at the last token can also store factual information in weight changes, not only MLPs.

Limitations

• OCR effects are sensitive to learning rate and number of epochs, and weaker in models other than OLMo-7b.
• Evaluation is restricted to synthetic two-hop datasets with a small number of entities (20 names, 20 city pairs).
• Informative and downstream metrics correlate only weakly (small Pearson correlation), and an unidentified alternative mechanism for OCR is suggested but not characterized.

Experimental setup

Datasets: First-hop synthetic dataset Second-hop synthetic dataset Fictitious dax/wug implications dataset

Benchmarks: First-hop synthetic OCR dataset Second-hop synthetic OCR dataset Fictitious dax/wug implication dataset

Metrics: mean rank of correct continuation extractive scores (informative, upstream, downstream)

Baselines: Facts first data ordering Joint data ordering Impl-first data ordering Control weight graft

Builds on: latent two-hop reasoning mean ablation influence functions

Closest papers by meaning

• Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.87)
• How Do Large Language Models Acquire Factual Knowledge During Pretraining? (NeurIPS 2024 · similarity 0.87)
• Delving into the Reversal Curse: How Far Can Large Language Models Generalize? (NeurIPS 2024 · similarity 0.87)
• How new data permeates LLM knowledge and how to dilute it (ICLR 2025 · similarity 0.87)
• Understanding Finetuning for Factual Knowledge Extraction (ICML 2024 · similarity 0.86)
• What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.86)