Research map / Detection and segmentation / Open-set and few-shot detection
Optimizing Relevance Maps of Vision Transformers Improves Robustness
Hila Chefer, Idan Schwartz, Lior Wolf
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize ViT relevance maps to emphasize foreground and suppress background | ViT-B · SI-location | R@1: 33.3 -> 38.6 | Improved |
| Add the confidence-boosting classification loss (ablation) | ViT-B finetuning · INet-A | top-1 accuracy: 17.9 -> 24.1 | Improved |
| Add the confidence-boosting classification loss (ablation) | ViT-B finetuning · ImageNet val | top-1 accuracy: 77.8 -> 80.3 | Improved |
| Add the background relevance loss (ablation) | ViT-B finetuning · INet-A | top-1 accuracy: 19.1 -> 24.1 | Improved |
| Add the background relevance loss (ablation) | ViT-B finetuning · ImageNet val | top-1 accuracy: 81.5 -> 80.3 | Hurt |
| Add the foreground relevance loss (ablation) | ViT-B finetuning · INet-A | top-1 accuracy: 24.1 -> 24.1 | No clear effect |
| Add the foreground relevance loss (ablation) | ViT-B finetuning · ImageNet val | top-1 accuracy: 80.2 -> 80.3 | Improved |
| Replace ground-truth-label cross-entropy with confidence boosting (ablation) | ViT-B finetuning · INet-A | top-1 accuracy: 21.5 -> 24.1 | Improved |
| Add the confidence-boosting classification loss (ablation) | DeiT-B finetuning · INet-A | top-1 accuracy: 15.5 -> 17.2 | Improved |
| Add the confidence-boosting classification loss (ablation) | DeiT-B finetuning · ImageNet val | top-1 accuracy: 81.0 -> 80.5 | Hurt |
| Add the background relevance loss (ablation) | DeiT-B finetuning · INet-A | top-1 accuracy: 13.2 -> 17.2 | Improved |
| Add the foreground relevance loss (ablation) | DeiT-B finetuning · INet-A | top-1 accuracy: 17.9 -> 17.2 | No clear effect |
Closest papers by meaning
- Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness (CVPR 2026 · similarity 0.89)
- Vision Transformers Are Robust Learners (AAAI 2022 · similarity 0.89)
- Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation (NeurIPS 2022 · similarity 0.88)
- Discrete Representations Strengthen Vision Transformer Robustness (ICLR 2022 · similarity 0.88)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.87)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
