Research map / Architectures and efficiency / Attention and transformers

Vision Transformers Don't Need Trained Registers

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: Theory of self-attention dynamics · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a sparse set of MLP "register neurons" causes high-norm outlier tokens in ViTs, and adds training-free test-time register tokens by moving those activations into an appended zero-initialized token, matching trained-register models on several vision and VLM tasks.

Problem

High-norm outlier tokens in pre-trained Vision Transformers (e.g., CLIP, DINOv2) create noisy attention maps and degrade downstream visual processing; the existing fix (register tokens) requires retraining models from scratch.

Approach

Identify register neurons in early-to-mid MLP layers whose activations concentrate on outlier patches, then at test time append an extra zero-initialized token and shift the register neurons' peak activations into it (zeroing them elsewhere), so outliers move outside the image without any training.

Key results

• On DINOv2 ViT-L/14, a test-time register raises unsupervised object discovery corloc on VOC07 from 32.2 to 53.8 (trained registers: 56.2).
• Zero-shot ImageNet segmentation mIoU for DINOv2 rises from 38.3 to 38.9 with a test-time register, above the trained-register model's 33.9.
• On LLaVA-Llama-3-8B, a test-time register keeps the eight-benchmark average at 46.2 versus baseline 46.2.

Measured effects reported in the paper

Surprising findings

• Zeroing out 10 register neurons in OpenCLIP ViT-B/16 drops zero-shot ImageNet accuracy from 70.4% to 55.6%, while ablating random neurons causes minimal change.
• Outliers are functionally essential: removing them rather than relocating them causes large performance drops, suggesting they must live in some token.

Limitations

• Only neuron-level contributions are edited; attention layers and positional encodings that also contribute to outliers are not addressed.
• Test-time registers are not fully equivalent to trained high-norm tokens, showing slight performance differences.
• Evaluated mainly on CLIP and DINOv2; other pretrained ViTs left for future work.

Experimental setup

Datasets: ImageNet LAION-2B LVD-142M ADE20k NYUv2 CIFAR-10 CIFAR-100 ImageNet-segmentation PASCAL VOC 2007 PASCAL VOC 2012 COCO 20k

Benchmarks: ImageNet ADE20k NYUv2 ImageNet-segmentation PASCAL VOC 2007 PASCAL VOC 2012 COCO 20k HallusionBench MMVet MMMU OCRBench MMStar

Metrics: top-1 accuracy mIoU rmse pixel accuracy mAP corloc

Baselines: DINOv2 ViT-L/14 with trained registers DINOv2 ViT-L/14 OpenCLIP ViT-B/16 OpenCLIP ViT-L/14 LOST LLaVA-Llama-3-8B

Builds on: Vision Transformers Need Registers Massive activations in LLMs Attention with bias / no-op attention LOST

Closest papers by meaning

• Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.91)
• Register and [CLS] tokens induce a decoupling of local and global features in large ViTs (NeurIPS 2025 · similarity 0.89)
• Vision Transformers Need Registers (ICLR 2024 · similarity 0.88)
• On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.87)
• No Head Left Behind – Multi-Head Alignment Distillation for Transformers (AAAI 2024 · similarity 0.87)
• Demystifying Singular Defects in Large Language Models (ICML 2025 · similarity 0.86)