Research map / Architectures and efficiency / Attention and transformers
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Yaru Hao, Li Dong, Furu Wei, Ke Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Insert attribution-derived triggers into test inputs | BERT-base-cased · MNLI entailment | accuracy: 82.87 -> 0.80 | Hurt |
| Insert attribution-derived triggers into test inputs | BERT-base-cased · SST-2 positive | accuracy: 92.79 -> 54.95 | Hurt |
| Insert attribution-derived triggers into test inputs | BERT-base-cased · MRPC equal | accuracy: 90.32 -> 29.39 | Hurt |
| Insert attribution-derived triggers into test inputs | BERT-base-cased · RTE entailment | accuracy: 72.60 -> 9.59 | Hurt |
| Rank heads using self-attention attribution scores rather than accuracy difference | BERT-base-cased head pruning · MNLI | accuracy | Improved |
| Retain heads with the highest attribution scores | BERT-base-cased · classification tasks | accuracy: nearly 97% | Improved |
| Use attribution-based head ranking instead of Taylor expansion ranking | BERT-base-cased head pruning · MNLI | accuracy: approximately 60%; about 40% | Mixed |
| Use attribution-derived adversarial patterns | BERT-based classifiers · MNLI, SST-2, MRPC, and RTE | accuracy | Hurt |
Closest papers by meaning
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
- AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers (ICML 2024 · similarity 0.87)
- Faithful and Accurate Self-Attention Attribution for Message Passing Neural Networks via the Computation Tree Viewpoint (AAAI 2025 · similarity 0.86)
- A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization (ICML 2025 · similarity 0.85)
- Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.85)
- Dissecting Query-Key Interaction in Vision Transformers (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
