Research map / Detection and segmentation / CNN features
Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Attention-Guided CAM, a gradient-based visualization for ViT that aggregates gradients along skip connections weighted by sigmoid-normalized self-attention, improving weakly-supervised localization and perturbation faithfulness over Attention Rollout and LRP.
Problem
Explaining ViT predictions with class-specific, high-semantic visual explanations; existing ViT methods (Attention Rollout, LRP) suffer from softmax peak intensities and scattered or non-class-specific heatmaps, hurting weakly-supervised localization.
Approach
Aggregate gradients from the class logit to each block's first-row self-attention via skip connections, and guide them with sigmoid-normalized attention scores as feature maps (instead of softmax) to reduce peak amplification, yielding a CAM for ViT.
Key results
- On ImageNet ILSVRC 2012, the method reaches 73.41% pixel accuracy, 52.12% IoU and 65.15% dice, versus 58.63% pixel accuracy for the LRP-based method.
- On PASCAL VOC 2012 it achieves 53.35% IoU and 81.67% precision, the highest among the three methods.
- In the pixel perturbation test on ILSVRC 2012, ABPC is 36.91% versus 34.04% for the LRP-based method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Attention Guided CAM instead of Attention Rollout | ViT-base · ImageNet ILSVRC 2012 val | IoU: 0.3597 -> 0.5212 | Helped |
| Use Attention Guided CAM instead of Attention Rollout | ViT-base · Pascal VOC 2012 val | IoU: 0.1645 -> 0.5335 | Helped |
| Use Attention Guided CAM instead of Attention Rollout | ViT-base · CUB 200 test | IoU: 0.3097 -> 0.5836 | Helped |
| Use Attention Guided CAM instead of Attention Rollout | ViT-base · ImageNet ILSVRC 2012 val | recall: 0.4657 -> 0.6276 | Helped |
| Use Attention Guided CAM instead of LRP-based CAM | ViT-base · ImageNet ILSVRC 2012 val | precision: 0.9110 -> 0.8299 | Hurt |
| Use Attention Guided CAM instead of LRP-based CAM | ViT-base · Pascal VOC 2012 val | precision: 0.7651 -> 0.8167 | Helped |
| Use Attention Guided CAM instead of LRP-based CAM | ViT-base · CUB 200 test | precision: 0.9669 -> 0.8987 | Hurt |
| Use Attention Guided CAM instead of LRP-based CAM | ViT-base · ImageNet ILSVRC 2012 | ABPC: 0.3404 -> 0.3691 | Helped |
Surprising findings
- The LRP-based method has higher precision than the proposed method on ILSVRC 2012 (91.10% vs 82.99%) and CUB 200 (96.69% vs 89.87%).
Limitations
- Precision is lower than the LRP-based method on several datasets, a trade-off for higher recall.
- The sigmoid-for-softmax gradient approximation relies on a smoothness assumption justified only heuristically in the supplement.
- Evaluation uses a single ViT-base model and excludes misclassified images from localization tests.
Experimental setup
Datasets: ImageNet ILSVRC 2012 PASCAL VOC 2012 CUB-200-2011
Benchmarks: ImageNet ILSVRC 2012 PASCAL VOC 2012 CUB-200-2011
Metrics: pixel accuracy IoU dice (F1) precision recall ABPC
Baselines: Attention Rollout LRP-based method for ViT raw attention Grad-CAM full-LRP partial-LRP
Builds on: Class Activation Map Attention Rollout Layer-wise Relevance Propagation Grad-CAM
Closest papers by meaning
- RegionViT: Regional-to-Local Attention for Vision Transformers (ICLR 2022 · similarity 0.89)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.87)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.87)
- LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.87)
- QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.