Research map / Label-efficient and robust learning / Self-supervised and contrastive
Attending on Attention ($A^2$): Smaller Self-Supervised ViTs Localize Better Than Larger Ones
Sreehari Rammohan, Huy Ha, Carl Vondrick
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows smaller self-supervised ViTs have attention maps that localize foreground objects better than larger ones, and proposes combining small-model localization with large-model patch representations for robust classification.
Problem
Robust visual classification often requires localizing main foreground objects while ignoring spatially separable distractors; it is unclear how attention localization quality varies with ViT size.
Approach
Smaller self-supervised ViTs yield better foreground-localizing attention than larger ones, while larger ViTs give richer per-patch representations; the method aims to combine good localization from small models with rich features from large models.
Key results
- Smaller self-supervised ViTs localize foreground objects better in their attention maps than larger ones (per the abstract's surprising finding).
Surprising findings
- Attention maps of smaller self-supervised ViTs localize foreground objects better than those of larger ViTs.
Closest papers by meaning
- Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention (AAAI 2024 · similarity 0.88)
- Contrastive Attention Maps for Self-Supervised Co-Localization (ICCV 2021 · similarity 0.88)
- CASTing Your Model: Learning To Localize Improves Self-Supervised Representations (CVPR 2021 · similarity 0.88)
- Register and [CLS] tokens induce a decoupling of local and global features in large ViTs (NeurIPS 2025 · similarity 0.87)
- Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level (NeurIPS 2024 · similarity 0.86)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
