Research map / Architectures and efficiency / Attention and transformers
The Extra Tokens Matter: Disentangled Representation Learning with Vision Transformers
Maofeng Tang, Hairong Qi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add minimum volume constraint (MVC) to factor tokens (ablation) | XTRA · ImageNet-1K | KNN accuracy (%): 72.4 -> 84.2 | Improved |
| add factor representation without MVC (ablation) | frozen teacher baseline · ImageNet-1K | KNN accuracy (%): 76.0 -> 72.4 | Hurt |
| freeze DINOv2 teacher (ablation) | baseline · ImageNet-1K | KNN accuracy (%): 76.1 -> 76.0 | No clear effect |
| use multi-stage aggregation (at least 1 stage) instead of none (ablation) | XTRA · ImageNet-1K | KNN accuracy (%): 13.9 -> 84.2 (final 2-stage model) | Improved |
| add third aggregation stage over two (ablation) | XTRA · ImageNet-1K | KNN accuracy (%): KNN increases 0.9%, linear probing increases 0.2% | No clear effect |
| use hard (one-hot) assignment instead of soft softmax (ablation) | XTRA with MVC · ImageNet-1K | SEPIN@1: 1.52 -> 3.95 | Improved |
| replace MVC with VICReg-L patch decorrelation (ablation) | XTRA · PartImageNet val | Part mIoU: 46.8 -> 43.8 | Hurt |
| unfreeze more student blocks (final block to all) (ablation) | XTRA · ImageNet-1K | KNN accuracy (%): 78.3 -> 84.2 | Improved |
| train XTRA with factor tokens on top of DINOv2 teacher | DINOv2 · ImageNet-C | mCE: 41.3 -> 38.2 | Improved |
| apply XTRA on different teachers (ablation) | MAE / CLIP / DINOv2 · ImageNet-1K | KNN accuracy (%): 65.2 -> 68.4 (MAE); 75.8 -> 79.3 (CLIP) | Improved |
| add XTRA without pretrained teacher (from scratch) | DINOv2 ViT-B · ImageNet-1K (from scratch) | KNN accuracy (%): 76.9 -> 81.9 | Improved |
| add register tokens without regularization (DINOv2+) | DINOv2 · ImageNet-1k | SEPIN@1: 0.47 -> 0.42 | No clear effect |
Closest papers by meaning
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.87)
- ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.87)
- A More Word-like Image Tokenization for MLLMs (CVPR 2026 · similarity 0.87)
- UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.87)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.87)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
