Research map / Architectures and efficiency / Attention and transformers
AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends Layer-wise Relevance Propagation with new attention-aware rules for softmax, matrix multiplication and normalization to attribute both inputs and latent neurons in transformers efficiently.
Problem
Faithful, efficient attribution of the full black-box transformer, including latent representations, is unsolved; existing LRP variants mishandle softmax, bilinear matrix multiplication and normalization, causing instability or low faithfulness.
Approach
Derives LRP propagation rules within the Deep Taylor Decomposition framework for non-linear attention: a Taylor rule with bias for softmax, an epsilon plus uniform rule for query-key/attention-value matrix multiplication, and identity rule for normalization; also uses the gamma-rule for ViT linear layers.
Key results
- AttnLRP achieves the highest faithfulness across ViT-B-16 (6.19), LLaMa 2-7b (2.50 IMDB, 10.93 Wikipedia), Mixtral 8x7b (0.96) and Flan-T5-XL (0.94) among compared methods.
- On Mixtral 8x7b, AttnLRP shows about 46% relative top-1 accuracy improvement over CP-LRP on SQuAD v2.
- Using AttnLRP on Phi-1.5 allows identifying knowledge neurons, and deactivating/amplifying them changes next-token output (e.g., Arctic to desert or sweets).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AttnLRP instead of KernelSHAP | ViT-B-16 · ImageNet | faithfulness: 4.71 -> 6.19 | Helped |
| Use AttnLRP instead of CP-LRP (ε-rule) | LLaMa 2-7b · IMDB | faithfulness: 1.72 -> 2.50 | Helped |
| Use AttnLRP instead of Grad×AttnRoll | LLaMa 2-7b · Wikipedia | faithfulness: 9.79 -> 10.93 | Helped |
| Use AttnLRP instead of Grad×AttnRoll | Mixtral 8x7b · SQuAD v2 | faithfulness: 0.91 -> 0.96 | Helped |
| Use AttnLRP instead of AtMan | Mixtral 8x7b · SQuAD v2 | top-1 accuracy: 0.86 -> 0.96 | Helped |
| Use AttnLRP instead of AtMan | Mixtral 8x7b · SQuAD v2 | IoU: 0.83 -> 0.72 | Hurt |
| Use AttnLRP instead of Grad×AttnRoll | Flan-T5-XL · SQuAD v2 | faithfulness: 0.94 -> 0.94 | No clear effect |
| Use AttnLRP instead of CP-LRP (ε-rule) | Flan-T5-XL · SQuAD v2 | IoU: 0.83 -> 0.84 | Helped |
| Replace the ε-rule with the γ-rule in ViT linear layers (ablation) | CP-LRP on ViT-B-16 · ImageNet | faithfulness: 2.53 -> 6.06 | Helped |
Surprising findings
- Distributing the softmax bias uniformly or applying the identity rule leads to relevance explosions to about 1e15 magnitudes, producing noise-like heatmaps.
- Gradient-based methods such as SmoothGrad and Integrated Gradients perform poorly on large transformers despite their popularity.
Limitations
- The gamma parameter in ViTs must be tuned per model and dataset to get accurate attributions.
- Memory consumption and quantization effects on attributions are not investigated; custom GPU kernels for LRP rules remain open.
Experimental setup
Datasets: ImageNet IMDB Wikipedia SQuAD v2
Benchmarks: ImageNet IMDB Wikipedia SQuAD v2
Metrics: faithfulness (area between perturbation curves) top-1 accuracy IoU
Baselines: Input x Gradient Integrated Gradients SmoothGrad GradCAM Attention Rollout Grad x AttnRoll AtMan KernelSHAP CP-LRP
Builds on: Layer-wise Relevance Propagation Deep Taylor Decomposition Conservative Propagation LRP Attention Rollout AtMan
Closest papers by meaning
- Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability (NeurIPS 2025 · similarity 0.91)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.89)
- Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens (NeurIPS 2024 · similarity 0.88)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.88)
- Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.88)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.