Research map / Architectures and efficiency / Attention and transformers
Thicker and Quicker: The Jumbo Token for Fast Plain Vision Transformers
Anthony Fuller, Yousef Yassin, Daniel Kyrollos, Evan Shelhamer, James R Green
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Jumbo token instead of registers | ViT-Nano · ImageNet-1K val (224px) | top-1 (%): 61.0 -> 74.6 | Improved |
| add Jumbo token instead of registers, larger width | ViT-Base · ImageNet-1K val (224px) | top-1 (%): 84.3 -> 84.9 | Improved |
| add Jumbo token instead of registers | ViT-Small · ImageNet-21K | top-1 (%): 41.5 -> 44.61 | Improved |
| add Jumbo to MAE pretraining | ViT-Base · ImageNet-1K linear probe | top-1 (%): 68.1 -> 73.0 | Improved |
| add Jumbo for test-time adaptation with SAR | ViT-Small · ImageNet-C | avg accuracy (%): 54.9 -> 60.1 | Improved |
| add Jumbo to fine-tuning for segmentation | ViT-Base · ADE20K | mIoU (%): 42.5 -> 44.4 | Improved |
| do not share Jumbo FFN across layers (ablation) | Jumbo ViT-Small · ImageNet-21K | top-1 (%): 44.61 -> 44.95 | Improved |
| add layer-specific LoRA to shared Jumbo FFN (ablation) | Jumbo ViT-Small · ImageNet-21K | top-1 (%): 44.61 -> 44.94 | Improved |
| add Jumbo FFN (ablation) | Jumbo ViT-Small · ImageNet-21K | top-1 (%): 43.64 -> 44.61 | Improved |
| increase Jumbo width multiplier J from 6 to 10 (ablation) | Jumbo ViT-Small · ImageNet-21K | top-1 (%): 44.61 -> 45.62 | Improved |
| widen registers ViT-Base to match Jumbo parameters (ablation) | ViT-Base+Registers · ImageNet-21K | top-1 (%): 45.7 -> 47.1 | No clear effect |
| add separate FFN for registers (ablation) | ViT-Small+Registers · ImageNet-21K | top-1 (%): 41.5 -> 42.1 | No clear effect |
Closest papers by meaning
- Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (AAAI 2022 · similarity 0.89)
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration (NeurIPS 2021 · similarity 0.89)
- Quick ViTs: Speeding up Vision Transformers through Equivariance (ECCV 2026 · similarity 0.89)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.88)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.87)
- A General and Efficient Training for Transformer via Token Expansion (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
