Research map / Architectures and efficiency / Attention and transformers
All Tokens Matter: Token Labeling for Training Better Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| token labeling instead of ReLabel supervision (ablation) | LV-ViT-S with MixToken · ImageNet-1k | top-1 accuracy: 83.0 -> 83.3 | Helped |
| MixToken instead of CutMix (ablation) | LV-ViT-S · ImageNet-1k | top-1 accuracy: 82.8 -> 83.0 | Helped |
| add MixToken augmentation (ablation) | LV-ViT-S with token labeling · ImageNet-1k | top-1 accuracy: 81.3 -> 83.3 | Helped |
| add MixUp on top of MixToken (ablation) | LV-ViT-S · ImageNet-1k | top-1 accuracy: 83.3 -> 83.1 | Hurt |
| add CutOut (ablation) | LV-ViT-S · ImageNet-1k | top-1 accuracy: 83.0 -> 83.3 | Helped |
| add RandAug (ablation) | LV-ViT-S · ImageNet-1k | top-1 accuracy: 82.8 -> 83.3 | Helped |
| use all tokens instead of 20% for token labeling loss (ablation) | LV-ViT-S · ImageNet-1k | top-1 accuracy: +0.5% | Helped |
| stronger annotator (EfficientNet-L2 JFT) instead of NFNet-F6 (ablation) | LV-ViT-S · ImageNet-1k | top-1 accuracy: 83.3 -> 83.5 | Helped |
| pretrain with token labeling (ablation) | LV-ViT-S + UperNet · ADE20K | mIoU (MS): 47.6 -> 48.6 | Helped |
| add token labeling | Mixer-L/16 · ImageNet-1k | top-1 accuracy: 77.7 -> 80.1 | Helped |
| 4-layer conv patch embedding instead of single linear patch embedding (ablation) | 16-layer ViT · ImageNet-1k | top-1 accuracy: 81.1 -> 82.2 | Helped |
| 6 convs instead of 4 in patch embedding (ablation) | 16-layer ViT · ImageNet-1k | top-1 accuracy: 82.2 -> 82.2 | No clear effect |
Closest papers by meaning
- ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.91)
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.91)
- Token Cropr: Faster ViTs for Quite a Few Tasks (CVPR 2025 · similarity 0.90)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.89)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.89)
- Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.