Research map / Architectures and efficiency / Attention and transformers
Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies low-rank communication channels between attention heads in GPT-2 small and Pythia-160m using SVD-decomposed composition scores, and shows an inhibition-channel subspace controls positional indexing that explains prompt-order sensitivity on the Laundry List task.
Problem
How transformer LMs represent and route information between layers, and what mechanism explains order-dependent failures when recalling items from long lists in context.
Approach
Decomposes attention-head OV and QK matrices via SVD into rank-1 components and uses a composition score on those components to find low-rank channels between heads separated by layers, then verifies them with weight editing and activation interventions; the inhibition channel's 3D subspace is shown to index list positions.
Key results
- Steering the top three inhibition components (7.9.6, 8.6.2, 8.6.10) raises 3-object Laundry List accuracy from 64% to 86%.
- GPT-2 small achieves FLDD of 97.5% and interchange accuracy of 35% on IOI inhibition channel patching.
- Removing the top-composing component of inhibition heads reduces inhibition score by 7-14%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Set the top three inhibition components according to the correct answer index | GPT-2 Small · Laundry List, 3 objects | accuracy: 64% to 86% | Helped |
| Set the top three inhibition components according to the correct answer index | GPT-2 Small · Laundry List, 8 objects | accuracy: about 51% | Helped |
| Patch inhibition-channel subspaces between minimally different examples | GPT-2 Small · IOI | FLDD: -8% to 97.5% | Helped |
| Patch inhibition-channel subspaces between minimally different examples | GPT-2 Small · IOI | interchange accuracy: 0.0% to 35% | Helped |
| Optimize a single vector directly for the task instead of using the discovered subspace | GPT-2 Small · IOI | FLDD: 111.5% | Helped |
| Optimize a single vector directly for the task instead of using the discovered subspace | GPT-2 Small · IOI | interchange accuracy: 45.1% | Helped |
| Use head 8.3 attention to distinguish likely from unlikely verb–object pairs | GPT-2 Small head 8.3 · Verb–object counterfactual pairs | attention score to noun: around 5% to around 35% | Helped |
| Add -1000 times the 8.3.0 output vector to the residual stream | GPT-2 Small head 9.9 · OpenWebText-10K excerpts | attention score on the token attended to by head 8.3: 9% to 12% | Mixed |
| Zero the highest-composition component in an inhibition head (ablation) | GPT-2 Small inhibition heads · IOI | inhibition score: reduces the Inhibition score by 7-14% | Hurt |
| Zero the selected component in head 7.3 (ablation) | GPT-2 Small head 7.3 · IOI | inhibition score | No clear effect |
| Keep only the top-1 composing components affecting inhibition (ablation) | GPT-2 Small inhibition heads · IOI | inhibition score: 0.7 to 0.3 | Hurt |
| Keep only the 0th singular-value components of inhibition heads (ablation) | GPT-2 Small inhibition heads · IOI | inhibition score: 0.04 | Hurt |
Surprising findings
- Interventions setting the inhibition component output to a scaled singular vector encode positional (first vs second name) inhibition independent of the context content.
- Composition scores on raw full matrices are noisy and fail to predict known head interactions, while SVD-decomposed components reveal outliers.
Limitations
- Relies on GPT-2 small, which is extensively studied, so insights may not generalize to other models.
- Query and key composition scores cannot be computed cleanly for models with RoPE due to non-linearities.
- Effect on larger models is not fully characterized; mechanisms in larger models may differ.
Experimental setup
Datasets: IOI dataset Laundry List synthetic task (250 prompts per N from 3 to 20) OpenWebText-10K counterfactual verb-noun pairs (84 pairs)
Benchmarks: Indirect Object Identification (IOI) Laundry List
Metrics: inhibition score fractional logit difference decrease interchange accuracy accuracy composition score
Baselines: unDecomposed composition score random head interventions subspace gradient optimization (Wang et al.)
Builds on: composition score indirect object identification circuit path patching logit lens
Closest papers by meaning
- Circuit Component Reuse Across Tasks in Transformer Language Models (ICLR 2024 · similarity 0.90)
- Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.89)
- Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism (ICML 2025 · similarity 0.89)
- Finding Interpretable Prompt-Specific Circuits in Language Models (NeurIPS 2026 · similarity 0.88)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.88)
- Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.