Research map / Architectures and efficiency / Attention and transformers
Causal Interpretation of Self-Attention in Pre-Trained Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLEANN instead of Smart-Attention to find sentiment explanations | BERT fine-tuned for sentiment classification · IMDB, review length 20 | explanation set size: 3.62 -> 2.82 | Helped |
| Use CLEANN instead of Smart-Attention to find sentiment explanations | BERT fine-tuned for sentiment classification · IMDB, review length 30 | explanation set size: 4.63 -> 3.48 | Helped |
| Use CLEANN instead of Smart-Attention to find sentiment explanations | BERT fine-tuned for sentiment classification · IMDB, review length 40 | explanation set size: 6.03 -> 3.58 | Helped |
| Use CLEANN instead of Smart-Attention to find sentiment explanations | BERT fine-tuned for sentiment classification · IMDB, review length 60 | explanation set size: 6.41 -> 3.53 | Helped |
| Use CLEANN instead of Pure-Attention to find sentiment explanations | BERT fine-tuned for sentiment classification · IMDB, review lengths 20, 30, 40, and 60 | explanation set size: 5.52 -> 2.82; 8.88 -> 3.48; 12.71 -> 3.58; 13.14 -> 3.53 | Helped |
| Use CLEANN instead of the attention-based baselines to explain recommendations | BERT4Rec · MovieLens 1M recommendation sessions | explanation set size | Helped |
| Use CLEANN instead of the attention-based baselines to select replacement recommendations | BERT4Rec · MovieLens 1M recommendation sessions, original top-5 list | replacement recommendation rank | Helped |
Closest papers by meaning
- How Transformers Learn Causal Structure with Gradient Descent (ICML 2024 · similarity 0.88)
- Towards Understanding the Universality of Transformers for Next-Token Prediction (ICLR 2025 · similarity 0.87)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
- Selective Induction Heads: How Transformers Select Causal Structures in Context (ICLR 2025 · similarity 0.86)
- Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers (NeurIPS 2025 · similarity 0.86)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.