Research map / Architectures and efficiency / Attention and transformers
Attention Meets Post-hoc Interpretability: A Mathematical Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mathematically analyzes a single-layer multi-head attention classifier to derive closed-form gradient and LIME explanations and compares them with attention-based explanations.
Problem
Whether attention weights can serve as faithful explanations of transformer predictions, and how they relate to post-hoc explanations (gradient-based and LIME), which has been debated mainly through empirical experiments without theoretical grounding.
Approach
Derives explicit expressions for gradient-based explanations (Gradient, Gradient x Input) of a single-layer multi-head attention classifier and asymptotic limit coefficients for LIME, showing how each depends on attention, value vectors, and the output linear layer, and contrasting these with raw attention-based explanations.
Key results
- Theorem-based LIME limit coefficients match official LIME outputs with average norm-2 error 0.808 (std 0.219) over the test set.
- A 6-layer 6-head classifier trained on IMDb reaches 82.22% accuracy; first-layer alpha-avg correlates with LIME coefficients (|rho| = 0.672 on absolute values).
- Gradient explanations contain a term linear in attention, but also depend on the output layer and value matrices, so attention alone does not capture them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add skip connections (ablation) | Single-layer attention classifier · IMDb | performance | No clear effect |
Surprising findings
- Attention-based explanations are always non-negative and cannot distinguish positively from negatively contributing tokens, unlike gradient x input and LIME.
Limitations
- Analysis restricted to a single-layer, multi-head model without skip connections or non-linearities beyond attention.
- LIME approximation assumes all tokens are distinct and holds only for large document and window sizes.
- Multi-layer experiments are preliminary and lack a formal dependency statement.
Experimental setup
Datasets: IMDb
Benchmarks: IMDb
Metrics: norm-2 error between LIME weights and theoretical approximat accuracy Pearson correlation
Baselines: attention-based explanations (alpha-avg, alpha-max) Gradient Gradient x Input LIME
Builds on: LIME Transformer Attention is not explanation debate (Jain & Wallace)
Closest papers by meaning
- What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.88)
- Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers (NeurIPS 2025 · similarity 0.86)
- Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention (NeurIPS 2026 · similarity 0.86)
- Consensus Is All You Get: The Role of Attention in Transformers (ICML 2025 · similarity 0.86)
- Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few (NeurIPS 2025 · similarity 0.86)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.