Research map / Large language models / Retrieval and knowledge
Knowledge Circuits in Pretrained Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses circuit discovery (ACDC-style zero ablation with a MatchNLL metric) to extract knowledge circuits of attention heads and MLPs in GPT-2 and TinyLLaMA, and uses them to analyze knowledge editing, multi-hop reasoning, hallucination and in-context learning.
Problem
Prior knowledge-storage work localizes facts to isolated MLP neurons or single components, which overlooks how attention heads and MLPs cooperate to store and express knowledge and limits understanding of knowledge editing failures and behaviors like hallucination.
Approach
Define a "knowledge circuit" as a subgraph of the residual-rewritten computation graph (attention heads, MLPs, embeddings) that is sufficient to answer a factual prompt, discovered by ablating edges and keeping only those whose removal hurts prediction; then analyze mover, relation and other heads inside it and how editing or failures change them.
Key results
- On GPT2-Medium, a circuit with under 10% of the edges retains over 70% of original performance (average test Hit@10 0.47 for circuit vs 0.44 original vs 0.00 random).
- Some relations improve inside the circuit, e.g. Landmark country rises from 0.16 to 0.36 and name birthplace from 0.07 to 0.57.
- Ablating newly appeared induction-like heads in the ICL circuit drops probability substantially (e.g. adj_comparative 62.24 to 32.55).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retain the discovered knowledge circuit instead of the full model | GPT2-Medium · Knowledge circuits, D_test (Avg) | Hit@10: 0.44 -> 0.47 | Helped |
| use the discovered knowledge circuit rather than a same-size random circuit | GPT2-Medium · Knowledge circuits, D_test (Avg) | Hit@10: 0.00 -> 0.47 | Helped |
| ablate the newly appeared ICL attention head (ablation) | GPT2-Medium ICL circuit · adj_comparative | Performance: 62.24 -> 32.55 | Hurt |
| ablate the newly appeared ICL attention head (ablation) | GPT2-Medium ICL circuit · word_sentiment | Performance: 89.02 -> 55.50 | Hurt |
| ablate the newly appeared ICL attention head (ablation) | GPT2-Medium ICL circuit · substance_phase | Performance: 78.74 -> 52.85 | Hurt |
| ablate the newly appeared ICL attention head (ablation) | GPT2-Medium ICL circuit · occupation_gender | Performance: 86.97 -> 59.54 | Hurt |
| ablate the newly appeared ICL attention head (ablation) | GPT2-Medium ICL circuit · person_occupation | Performance: 35.17 -> 23.27 | Hurt |
Surprising findings
- Circuit-only evaluation sometimes outperforms the full original model on several relations, suggesting noise from other components.
- Knowledge circuits show a circuit reuse phenomenon where related relations share heads such as L21H12.
Limitations
- Circuit discovery via ablation/patching is time-intensive and coarse-grained (components, not neurons).
- Logit lens may mismatch intermediate-layer representations with the unembedding matrix.
- Analysis is limited to GPT-2 and TinyLLaMA with a small number of relations and examples.
Experimental setup
Datasets: LRE dataset (factual, commonsense, linguistic, bias relations)
Benchmarks: LRE relation dataset subsets (Hit@10 completeness)
Metrics: Hit@10 rank change across layers node/edge overlap
Baselines: ROME FT-M random circuit
Builds on: ACDC automated circuit discovery ROME knowledge neurons LRE relation dataset logit lens
Closest papers by meaning
- Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons (AAAI 2024 · similarity 0.89)
- Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshaping (ICLR 2026 · similarity 0.88)
- Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing (ICML 2025 · similarity 0.88)
- What does the Knowledge Neuron Thesis Have to do with Knowledge? (ICLR 2024 · similarity 0.88)
- Circuit Component Reuse Across Tasks in Transformer Language Models (ICLR 2024 · similarity 0.87)
- Localizing Knowledge in Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.