Research map / Architectures and efficiency / Attention and transformers
Vision Transformers with Self-Distilled Registers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PH-Reg, a post-hoc self-distillation method that adds learnable register tokens to a pretrained ViT, using a frozen teacher denoised via test-time shift/flip augmentation averaging, to remove artifact tokens without full retraining.
Problem
Artifact tokens in pretrained Vision Transformers degrade fine-grained dense prediction, but adding register tokens normally requires costly retraining from scratch, which is infeasible for large existing pretrained ViTs.
Approach
Initialize a frozen teacher and a student from the same pretrained ViT; the student gets randomly initialized registers and a few unlocked weights (registers, positional embeddings, patch embedding, last attention block). The teacher's targets are denoised by averaging features over 10 shifted/flipped augmented views (a gradient-free alternative to DVT's neural-field denoiser), and the student is
Key results
- Zero-shot open-vocabulary segmentation: PH-Reg (CLIP) averages 41.85 mIoU over 8 benchmarks vs 40.08 for SCLIP and 39.91 for NACLIP+DVT, best on 7 of 8 datasets.
- Linear probe: DINOv2 + Ours reaches 0.1216 Abs Rel and 86.35 delta1 on NYUd, improving over DINOv2 (0.1391, 82.92); CLIP + Ours gains +5.04 mIoU on VOC21 (75.32 vs 70.28 for MaskCLIP baseline in table).
- Training time on a single GPU for DINOv2 ViT-B: about 9000 min for PH-Reg vs 21908 min for DVT, saving over 58.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PH-Reg self-distillation with registers and teacher denoising | OpenAI CLIP ViT-B/16 · Open-vocabulary segmentation, 8-benchmark average | mIoU (%): 32.21 -> 41.85 | Helped |
| add register tokens to the distilled student (ablation) | OpenAI CLIP ViT-B/16 · Open-vocabulary segmentation, 8-benchmark average | mIoU (%): 41.48 -> 41.85 | Helped |
| add denoising to the teacher (ablation) | OpenAI CLIP ViT-B/16 · Open-vocabulary segmentation, 8-benchmark average | mIoU (%): 40.66 -> 41.85 | Helped |
| use 10x test-time augmentation denoising without distillation (ablation) | MaskCLIP · Open-vocabulary segmentation, 8-benchmark average | mIoU (%): 32.21 -> 34.55 | Helped |
| increase register count from 4 to 8, then 8 to 16 (ablation) | PH-Reg student · COCO Caption | cosine similarity | No clear effect |
| unlock positional embeddings during distillation (ablation) | PH-Reg student · COCO Caption | cosine similarity | Helped |
| unlock the final attention layer (ablation) | PH-Reg student · COCO Caption | cosine similarity | Helped |
| unlock the convolutional patch embedding layer alone (ablation) | PH-Reg student · COCO Caption | cosine similarity | Hurt |
| unlock the convolutional patch embedding together with later attention layers (ablation) | PH-Reg student · COCO Caption | cosine similarity | Helped |
| increase teacher augmentations from 1 to 10 (ablation) | PH-Reg denoising · COCO Caption | cosine similarity | Helped |
| set shifting ratio to 15% rather than 0% (ablation) | PH-Reg teacher denoising · Open-vocabulary segmentation, 8-benchmark average | mIoU (%): 39.41 -> 40.27 | Helped |
| use PH-Reg instead of ClearCLIP | OpenAI CLIP ViT-B/16 · VOC20 | mIoU (%): 84.56 -> 83.05 | Hurt |
Surprising findings
- Artifact tokens are not always high-norm: in OpenAI CLIP and OpenCLIP artifacts are lower-norm than surrounding patches.
- Positional embeddings alone do not explain artifacts, unlike prior claims; DVT's static-artifact assumption fails for CLIP models, so DVT gives only small gains there.
Limitations
- Underperforms DVT on some DINOv2 settings, which the authors attribute to DVT's static-artifact estimator fitting DINOv2 well.
- Relies on test-time augmentation with 10 views to denoise teacher features, adding teacher-side compute during training.
- Evaluated mainly on ViT-B backbones and a limited set of dense tasks.
Experimental setup
Datasets: COCO Caption
Benchmarks: VOC21 PC60 Object VOC20 PC59 Stuff City ADE20K PASCAL VOC 2012 linear probe ADE20K linear probe NYUv2 depth
Metrics: mIoU mAcc RMSE Abs Rel delta1 Pearson correlation cosine similarity GFLOPs
Baselines: CLIP MaskCLIP SCLIP ClearCLIP NACLIP MaskCLIP + DVT NACLIP + DVT CLIP + DVT OpenCLIP DFN-CLIP DINOv2 DINOv2 + DVT
Builds on: Vision Transformers Need Registers Denoising Vision Transformers (DVT) MaskCLIP NACLIP
Closest papers by meaning
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.92)
- Vision Transformers Don't Need Trained Registers (NeurIPS 2025 · similarity 0.91)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.88)
- Register and [CLS] tokens induce a decoupling of local and global features in large ViTs (NeurIPS 2025 · similarity 0.88)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
- Attention Transfer Is Not Universally Effective for Vision Transformers (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.