Research map / Architectures and efficiency / Attention and transformers
Register and [CLS] tokens induce a decoupling of local and global features in large ViTs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how register tokens and the [CLS] skip connection in large DINOv2 ViTs decouple global [CLS] outputs from local patch features, showing attention maps over patches fail to reflect global representations.
Problem
Attention maps of large ViTs (DINOv2 with and without registers) are commonly assumed to show which patches yield the global [CLS] representation, but this "patch integration assumption" may be violated in large models.
Approach
Shows that in large ViTs, the [CLS] output is dominated by register tokens (or the [CLS] skip connection) rather than patch tokens, using attention partitioning, CKA between global and patch-only/register-only outputs, and one-shot classification; argues the [CLS] residual acts as an implicit register.
Key results
- In the DINOv2 giant model with registers, the [CLS] token attends primarily to register tokens, and the register-based output represents the global output more faithfully than patch-based output by CKA.
- Cosine similarity between the patch-based [CLS] output and the full output in the giant model with registers is -0.0092, showing the attention map fails to attribute global information to patches.
- Without registers, the giant DINOv2 [CLS] output is dominated by skip-connection features, and CKA between patch-based output and global output is low for large models while near perfect for smaller models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from small/base to large/giant | DINOv2 with register tokens · MS COCO validation images | last-layer [CLS] attention to registers | Mixed |
| Increase model size | DINOv2 with register tokens · MS COCO test set | CKA of patch-only output with full [CLS] output | Hurt |
| Restrict final-layer attention to register tokens | DINOv2 giant with registers · MS COCO test set | CKA with full [CLS] output | Helped |
| Restrict final-layer attention to patch tokens | DINOv2 giant with registers · Example image | cosine similarity to full output: -0.0092 | Hurt |
| Use patch-only rather than full output for evaluation | DINOv2 large/giant with registers · ImageNet one-shot classification, top-5 accuracy | accuracy | Hurt |
| Increase model size | DINOv2 without registers · MS COCO validation set | relative contribution of [CLS] skip features | Hurt |
| Restrict [CLS] output to patch features rather than skip features | DINOv2 giant without registers · MS COCO | CKA with full model output | Hurt |
| Add drop-path regularization (ablation) | DeiT III · Patch integration assumption | local-global feature correspondence | No clear effect |
| Increase model size | DINOv2 without registers · MS COCO | CKA of patch-only output with full output | Hurt |
| Use the penultimate-layer register token representation | DINOv2 giant with registers · Across images | pairwise cosine similarity: 0.9989 | No clear effect |
Surprising findings
- Layer norm downscales the high-norm register dimensions, so near-identical cosine similarity (0.9989) across images does not imply image-independent register information.
- Cosine similarity between patch-only and full [CLS] output is -0.0092 in the giant model with registers, i.e., orthogonal.
Limitations
- Analysis is restricted to pretrained off-the-shelf models; no controlled retraining without registers or skip connections was performed due to compute cost.
- Does not study the impact of these violations on global task performance.
- Skip-connection attention is inferred only via L2-norm proxies since attention weights are implicit.
Experimental setup
Datasets: MS COCO ImageNet
Benchmarks: MS COCO validation ImageNet test
Metrics: attention mass linear CKA one-shot top-5 accuracy cosine similarity L2-norm contribution
Baselines: DINOv2 with registers DINOv2 without registers
Builds on: DINOv2 Vision Registers (Darcet et al.) Sun et al. high-norm tokens analysis
Closest papers by meaning
- Vision Transformers Don't Need Trained Registers (NeurIPS 2025 · similarity 0.89)
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.89)
- To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.89)
- Mamba-Reg: Vision Mamba Also Needs Registers (CVPR 2025 · similarity 0.88)
- RATS! Patches Talk Through Registers: Emergent Parts in Register Attention Transformers (NeurIPS 2026 · similarity 0.88)
- See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.