Research map / Architectures and efficiency / Attention and transformers
Dissecting Query-Key Interaction in Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes ViT query-key interactions via SVD of W_q^T W_k, showing early layers attend to similar tokens (grouping) and late layers to dissimilar tokens (contextualization), with interpretable singular modes.
Problem
Whether self-attention in vision transformers mainly groups similar tokens or attends to dissimilar tokens for contextualization, and how query-key interactions can be interpreted semantically.
Approach
Decompose the query-key interaction matrix W_q^T W_k by SVD into singular modes, treating left and right singular vectors as paired feature directions; measure grouping vs. contextualization via cosine similarity between each mode's left and right singular vectors, weighted by singular values.
Key results
- In most ViTs, especially classification-trained ones, early layers attend more to same-object tokens while later layers increasingly attend to background and different objects.
- Some self-supervised ViTs (e.g., DINO, DeiT, SimMIM pretrain) show a concave trend, increasing attention to similar tokens in the last few layers.
- Singular modes are often semantically interpretable, e.g., part-to-part, object-to-object, and foreground-to-background interactions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from early to late layers | Most ViTs, especially classification-trained models · Odd-One-Out (O3) | attention allocation to same objects versus contextual tokens | Mixed |
| move from early to late layers | ViTs · ImageNet singular-mode analysis | weighted cosine similarity of left and right singular vectors | Mixed |
| use pixel-level mask-prediction pretraining rather than classification fine-tuning | SimMIM ViT-B/16 · O3 and layer-wise attention analysis | attention to same objects | Mixed |
| use a self-supervised objective rather than a classification objective | ViTs · Layer-wise query-key interaction analysis | attention to similar versus dissimilar tokens | Mixed |
| move from early to late layers | ViTs · ADE20K | probability that left and right singular vectors select the same object | Mixed |
Surprising findings
- Self-supervised ViTs such as DINO, DeiT, and SimMIM pretrain increase attention to similar tokens in their last layers, unlike classification models.
- SimMIM's trend of attending to similar tokens disappears after fine-tuning on classification.
Limitations
- Only query-key interactions are analyzed; the value matrix's role is not studied.
- Behavior varies considerably across models, and the link to training objectives is only hypothesized.
- Error bars are not drawn in main figures; confidence intervals appear only in supplementary distribution plots.
Experimental setup
Datasets: Odd-One-Out (O3) ImageNet validation set ADE20K
Benchmarks: Odd-One-Out (O3) ImageNet validation ADE20K
Metrics: attention overlap ratio with same/different object/backgroun singular-value-weighted cosine similarity between left and r probability left and right singular vectors highlight same o
Baselines: ViT DeiT CLIP DINO SimMIM I-JEPA
Builds on: Vision Transformer DINO Odd-One-Out dataset
Closest papers by meaning
- What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.88)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.87)
- Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.87)
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.87)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.87)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.