Research map / Architectures and efficiency / Attention and transformers
What Makes for Good Tokenizers in Vision Transformer?
Shengju Qian, Yi Zhu, Wenbo Li, Mu Li, Jiaya Jia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MoTo to DeiT-S tokenizer | DeiT-S · ImageNet-1k | top-1 accuracy: 79.8 -> 81.6 | Improved |
| add MoTo across ViT, T2T, PVT, Swin families | various transformers · ImageNet-1k | top-1 accuracy: +0.7 to +1.8 | Improved |
| add TokenProp reconstruction objective to tokenizer | DeiT-S · ImageNet-1k | top-1 accuracy: 79.8 -> 80.7 | Improved |
| combine MoTo and TokenProp | DeiT-S · ImageNet-1k | top-1 accuracy: 79.8 -> 82.6 | Improved |
| use soft instead of hard semantic partition in MoTo (ablation) | MoTo on DeiT-S · ImageNet-1k | top-1 accuracy: 81.7 -> 80.5 (n=16 soft vs hard) | Improved |
| increase semantic entities beyond n=8 (ablation) | MoTo on DeiT-S · ImageNet-1k | top-1 accuracy: 81.6 -> 81.7 | No clear effect |
| add LayerNorm or BatchNorm in tokenizer instead of MoTo (ablation) | DeiT-S · ImageNet-1k | top-1 accuracy: 79.8 -> 79.6 (LN), 79.5 (BN) | Hurt |
| add InstanceNorm in tokenizer (ablation) | DeiT-S · ImageNet-1k | top-1 accuracy: 79.8 -> 80.1 | No clear effect |
| absorb MoTo into transformer blocks 8-12 as well (ablation) | DeiT-S w MoTo · ImageNet-1k | top-1 accuracy: 81.6 -> 81.9 (blocks 1-8), 81.4 (all) | Mixed |
| use L2 reconstruction loss versus L1 in TokenProp (ablation) | TokenProp on DeiT-S · ImageNet-1k | top-1 accuracy: 80.2 -> 80.7 | Improved |
| raise reconstruction weight lambda to 1.0 (ablation) | TokenProp on DeiT-S · ImageNet-1k | top-1 accuracy: 80.7 -> NaN | Hurt |
| reconstruct at 256x256 instead of 64x64 (ablation) | TokenProp on DeiT-S · ImageNet-1k | top-1 accuracy: 80.7 -> 77.3 | Hurt |
Closest papers by meaning
- Frequency-Aware Token Reduction for Efficient Vision Transformer (NeurIPS 2025 · similarity 0.89)
- ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.88)
- Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer (CVPR 2024 · similarity 0.87)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.87)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.87)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
