Research map / Architectures and efficiency / Attention and transformers
Generic Attention-Model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers
Hila Chefer, Shir Gur, Lior Wolf
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add self-attention normalization to bi-modal rule (ablation) | explainability method on DETR · COCO val weak segmentation | AP: 11.7 -> 13.1 | Improved |
| add self-attention normalization to bi-modal rule (ablation) | explainability method on DETR · COCO val weak segmentation | AP_large: 20.9 -> 24.6 | Improved |
| add self-attention normalization (ablation) | explainability method on DETR · COCO val weak segmentation | AR_medium: 23.9 -> 23.9 | No clear effect |
| add aggregation of previous relevancy matrices (ablation) | explainability method on DETR · COCO val weak segmentation | AP: 0.1 -> 13.1 | Improved |
| add self-attention terms to bi-modal rule (ablation) | explainability method on DETR · COCO val weak segmentation | AP: 11.5 -> 13.1 | Improved |
| add normalization (ablation) | explainability method on LXMERT · VQA perturbation | negative AUC, image tokens: 62.49 -> 63.24 | Improved |
| add aggregation (ablation) | explainability method on LXMERT · VQA perturbation | negative AUC, image tokens: 60.41 -> 63.24 | Improved |
| add aggregation (ablation) | explainability method on LXMERT · VQA perturbation | positive AUC, image tokens (lower better): 60.40 -> 51.10 | Improved |
| add self-attention terms to bi-modal rule (ablation) | explainability method on LXMERT · VQA perturbation | negative AUC, image tokens: 62.18 -> 63.24 | Improved |
| add LRP to head averaging | explainability method on LXMERT · VQA perturbation | negative AUC, image tokens: 63.24 -> 62.41 | Hurt |
| add LRP to head averaging | explainability method on LXMERT · VQA perturbation | negative AUC, text tokens: 48.70 -> 48.25 | Hurt |
| use proposed method instead of Transformer attribution | DETR · COCO val weak segmentation | AP: 7.2 -> 13.1 | Improved |
Closest papers by meaning
- Mr. DETR: Instructive Multi-Route Training for Detection Transformers (CVPR 2025 · similarity 0.87)
- Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention (AAAI 2024 · similarity 0.87)
- Dissecting Query-Key Interaction in Vision Transformers (NeurIPS 2024 · similarity 0.87)
- On the Faithfulness of Vision Transformer Explanations (CVPR 2024 · similarity 0.86)
- Attention Meets Post-hoc Interpretability: A Mathematical Perspective (ICML 2024 · similarity 0.86)
- Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
