Research map / Large language models / Language, safety and interpretability
Explorations of Self-Repair in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that attention-head self-repair (downstream compensation after resample-ablating a head) occurs across the full pretraining distribution in many model families, attributes part of it to LayerNorm scale changes and sparse anti-erasure MLP neurons, and offers the Iterative Inference hypothesis as a speculative explanation.
Problem
Ablation-based interpretability metrics can be misleading because downstream components compensate for ablated components (self-repair); prior evidence was limited to narrow distributions or whole-layer ablations.
Approach
Measures self-repair of individual attention heads across the full pretraining distribution (The Pile) rather than narrow tasks, identifies LayerNorm scaling-factor changes as a non-trivial self-repair mechanism, and localizes MLP anti-erasure to sparse sets of neurons.
Key results
- Self-repair of attention heads occurs across many later layers of multiple models over 1M tokens of The Pile, but is imperfect and noisy at the token level.
- LayerNorm scale changes explain a nontrivial fraction of self-repair, around 30% of a head's direct effect on average (clipped fraction metric).
- For L11H2 in Pythia-160M, the clean/ablated LayerNorm scale ratio exceeds 1 on 90.66% of top-2% tokens for resample ablation, 98.75% for mean, and 100% for zero ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Resample-ablate individual attention heads | Pythia models and other model families and sizes · Full pretraining distribution; The Pile | self-repair of attention-head direct effects | Helped |
| Resample-ablate individual attention heads | Attention heads across models · Full pretraining distribution; The Pile | completeness and consistency of self-repair | Mixed |
| Allow the final LayerNorm scaling factor to change during head ablation | Ablated attention heads · The Pile | self-repair of direct effects: around 30% | Helped |
| Use mean ablation instead of resample ablation (ablation) | L11H2 in Pythia-160M · The Pile, top 2% of tokens by direct effect | tokens with LayerNorm scaling ratio greater than 1: 90.66% -> 98.75% | Helped |
| Use zero ablation instead of resample ablation (ablation) | L11H2 in Pythia-160M · The Pile, top 2% of tokens by direct effect | tokens with LayerNorm scaling ratio greater than 1: 90.66% -> 100% | Helped |
| Zero-ablate a head rather than resample-ablate it (ablation) | Attention heads with positive direct effect · The Pile | logit difference | Mixed |
| Freeze all downstream components, including the LayerNorm normalization factor (ablation) | Ablated attention heads · The Pile | recovered logit difference | Hurt |
| Remove an upstream head that induces downstream MLP erasure | Sparse erasure neurons in the final MLP layer · The Pile | self-repair of the head's direct effect | Helped |
| Resample-ablate L10H11 | Final-layer neurons in Pythia-160M · The Pile | direct effect of the top self-repairing neuron | Helped |
| Add the head's output back into the residual stream that feeds into the head | L21H1 in GPT2-Medium · The Pile | head direct effect | Hurt |
Surprising findings
- Zero ablation can increase the logit difference relative to the clean run even when the ablated head has positive direct effect, purely through LayerNorm changes.
- Many heads with significant direct effects show no clear correlation between direct effect and logit change, suggesting near-full self-repair in some heads.
Limitations
- Focuses on repair of direct effects rather than behavioral measures of head importance.
- LayerNorm explains only around 30% of self-repair on average; the remainder is unexplained by the analysis.
- Iterative Inference hypothesis is speculative and supported only by anecdotal evidence.
Experimental setup
Datasets: The Pile
Benchmarks: The Pile
Metrics: logit difference of correct next token direct effect self-repair (delta logit minus delta direct effect) LayerNorm scale ratio fraction of direct effect self-repaired
Baselines: zero ablation mean ablation
Builds on: Indirect Object Identification circuit analysis Hydra Effect Copy Suppression MLP Erasure Iterative Inference hypothesis
Closest papers by meaning
- A Theoretical Understanding of Self-Correction through In-context Alignment (NeurIPS 2024 · similarity 0.89)
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.87)
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.86)
- Interpreting the Repeated Token Phenomenon in Large Language Models (ICML 2025 · similarity 0.86)
- Coherence Mechanisms for Provable Self-Improvement (NeurIPS 2026 · similarity 0.86)
- Restoring Pruned Large Language Models via Lost Component Compensation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.