Research map / Architectures and efficiency / Attention and transformers
Vision Transformers Need Registers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies high-norm artifact tokens in ViT feature maps (DINOv2, DeiT-III, OpenCLIP) and removes them by appending learnable register tokens to the input sequence during training.
Problem
Large, long-trained vision transformers (DINOv2, supervised DeiT-III, OpenCLIP) develop high-norm outlier patch tokens in low-information background regions, producing noisy attention and feature maps that hurt dense prediction and object discovery.
Approach
Append a few learnable register tokens to the ViT input sequence, unused at the output, so the model stores global computation in them instead of recycling redundant patch tokens; this removes norm outliers and yields smoother features without degrading performance.
Key results
- Adding 4 registers to DINOv2 raises linear-probe ADE20k mIoU from 46.6 to 47.9 and lowers NYUd rmse from 0.378 to 0.366, with ImageNet top-1 rising from 84.3 to 84.8.
- On LOST object discovery, DINOv2 corloc on VOC 2007 improves from 35.3 to 55.4 with registers.
- Registers remove high-norm output patch tokens in DINOv2, OpenCLIP and DeiT-III; FLOP overhead for 4 registers is below 2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add register tokens | DINOv2 ViT-L · ADE20k linear probing | mIoU: 46.6 -> 47.9 | Helped |
| Add register tokens | DINOv2 ViT-L · NYUd linear probing | rmse: 0.378 -> 0.366 | Helped |
| Add register tokens | DINOv2 ViT-L · VOC 2007 object discovery | corloc: 35.3 -> 55.4 | Helped |
| Add register tokens | DINOv2 ViT-L · VOC 2012 object discovery | corloc: 40.2 -> 60.0 | Helped |
| Add register tokens | DINOv2 ViT-L · COCO 20k object discovery | corloc: 26.9 -> 42.0 | Helped |
| Add register tokens | DeiT-III ViT-B · VOC 2007 object discovery | corloc: 11.7 -> 27.1 | Helped |
| Add register tokens | DeiT-III ViT-B · VOC 2012 object discovery | corloc: 13.1 -> 32.7 | Helped |
| Add register tokens | DeiT-III ViT-B · COCO 20k object discovery | corloc: 10.7 -> 25.1 | Helped |
| Add register tokens | OpenCLIP ViT-B/16 · VOC 2007 object discovery | corloc: 38.8 -> 37.1 | Hurt |
| Add register tokens | OpenCLIP ViT-B/16 · COCO 20k object discovery | corloc: 31.0 -> 27.9 | Hurt |
| Add register tokens | OpenCLIP ViT-B/16 · ImageNet zero-shot classification | Top-1: 59.9 -> 60.1 | No clear effect |
| Vary the number of register tokens (0, 1, 2, 4, 8 or 16) (ablation) | DINOv2 ViT-L/14 · ImageNet, ADE-20k and NYUd downstream evaluation | performance | Mixed |
Surprising findings
- OpenCLIP's LOST object discovery performance slightly drops with registers (VOC 2007 38.8 to 37.1), despite artifacts being removed.
- Outlier tokens hold strong global information (linear probe on outlier patch tokens beats normal patches), suggesting models repurpose redundant background patches as global storage.
Limitations
- The cause of artifact emergence is not fully determined; pretraining paradigm, model size and training length all appear to matter.
- The high-norm cutoff (150) is hand-picked and may vary across models.
- Object discovery for OpenCLIP does not benefit, and DINOv2 with registers still does not match DINO on VOC 2007.
Experimental setup
Datasets: ImageNet-22k ImageNet-1k Shutterstock licensed image-text corpus PASCAL VOC 2007 PASCAL VOC 2012 COCO 20k ADE20k NYUd Aircraft CIFAR-10 CIFAR-100 CUB
Benchmarks: ImageNet ADE20k NYUd PASCAL VOC 2007 PASCAL VOC 2012 COCO 20k ImageNet zero-shot
Metrics: top-1 accuracy mIoU rmse corloc norm distribution linear probe accuracy
Baselines: DINOv2 DeiT-III OpenCLIP DINO LOST
Builds on: DINOv2 DINO LOST Memory Transformer DeiT-III
Closest papers by meaning
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.92)
- Vision Transformers Need More Than Registers (CVPR 2026 · similarity 0.91)
- Efficient Self-supervised Vision Transformers for Representation Learning (ICLR 2022 · similarity 0.91)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.89)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.88)
- Vision Transformers Don't Need Trained Registers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.