Research map / Large language models / Language, safety and interpretability
Linking In-context Learning in Transformers to Human Episodic Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Links induction heads in Transformer LLMs to the CMR model of human episodic memory, showing CMR-like asymmetric contiguity attention biases in intermediate-to-late layers and a causal ablation effect on ICL.
Problem
Whether and how the emergent behavior and mechanisms of interacting attention heads (induction heads) relate to human episodic memory, given that Transformers are rarely linked to biological cognition.
Approach
Shows that the CMR model of free recall can be rewritten as a two-layer Q-composition induction head, then quantifies per-head similarity to CMR via a CMR distance (fitted CRP-like attention lag profile) and finds CMR-like heads in intermediate-to-late layers that are causally relevant for in-context learning.
Key results
- Top CMR-like heads show average CMR distance 0.11 (top 20) vs Gaussian distance 1.0 across 12 models, with all p<0.0001.
- CMR-like heads (lowest CMR distance) concentrate in intermediate-to-late layers across GPT2-small, Pythia, Qwen-7B, Mistral-7B and Llama3-8B.
- Ablating the top 10% CMR-like heads generally worsens ICL score more than ablating the same number of random heads, most clearly for GPT2 and Qwen-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fit attention scores with CMR rather than a Gaussian function | Top 20 induction heads across 12 models · Top 20 induction heads | distance: 1.0 -> 0.11 | Helped |
| Fit attention scores with CMR rather than a Gaussian function | Top 50 induction heads across 12 models · Top 50 induction heads | distance: 0.98 -> 0.05 | Helped |
| Fit attention scores with CMR rather than a Gaussian function | Top 100 induction heads across 12 models · Top 100 induction heads | distance: 0.98 -> 0.12 | Helped |
| Fit attention scores with CMR rather than a Gaussian function | Top 200 induction heads across 12 models · Top 200 induction heads | distance: 0.97 -> 0.12 | Helped |
| Ablate the top 10% of CMR-like heads instead of the same number of randomly selected heads (ablation) | GPT2, Pythia models, and Qwen-7B · C4 sampled texts | ICL score | Hurt |
Surprising findings
- Some heads not traditionally classed as induction heads (e.g., duplicate token heads peaking at lag 0) are well captured by CMR.
- Pythia's ICL scores are closer to 0 than other models, suggesting weaker ICL or distribution mismatch.
Limitations
- Unclear whether CMR serves as a mechanistic model in larger Transformers.
- Hydra effect: ablation causes other heads to compensate, weakening causal interpretation.
- Causal analysis cannot isolate which CMR-like characteristics drive ICL; ICL score is a limited metric.
Experimental setup
Datasets: Google C4 (processed version) PEERS dataset (human free recall, reproduced from prior work)
Benchmarks: C4 ICL score (loss at 500th token minus loss at 50th token)
Metrics: CMR distance (MSE vs CMR-fitted CRP) induction-head matching score copying score ICL score percentage of heads with CMR distance below threshold
Baselines: Gaussian distance baseline
Builds on: contextual maintenance and retrieval model induction heads Q-composition TransformerLens
Closest papers by meaning
- What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation (ICML 2024 · similarity 0.89)
- Which Attention Heads Matter for In-Context Learning? (ICML 2025 · similarity 0.89)
- Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers (NeurIPS 2025 · similarity 0.89)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.89)
- Selective Induction Heads: How Transformers Select Causal Structures in Context (ICLR 2025 · similarity 0.88)
- Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.