Research map / Architectures and efficiency / Attention and transformers
ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inference
Ali Hojjat, Janek Haberer, Sören Pirk, Olaf Landsiedel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| progressively expand to more attention heads instead of looping the same model (ablation) | DeiT-Tiny · ImageNet-1K | top-1 accuracy (%): 74.10 (naive 2 extra iterations) vs 81.44 (ThinkingViT 3H->6H) | Improved |
| naive looping of same ViT with more iterations (ablation) | DeiT-Tiny · ImageNet-1K | top-1 accuracy (%): 72.20 -> 74.00 -> 74.10 -> 73.60 | Mixed |
| add a third thinking stage (12 heads) (ablation) | ThinkingViT 3H->6H · ImageNet-1K | accuracy / GMACs: +0.91 p.p.; 5.85 -> 23.41 GMACs | Mixed |
| add a 2H stage before 3H->6H (2H->3H->6H) (ablation) | ThinkingViT · ImageNet-1K | accuracy vs GMACs range | Mixed |
| add Token Recycling and progressive heads over flat 6-head models | ThinkingViT · ImageNet-1K | top-1 accuracy (%): up to +2.0 p.p. at equal throughput, +2.9 p.p. at equal GMACs | Improved |
| apply ThinkingViT to a Swin-S backbone | Swin-S · ImageNet-1K | GMACs at Swin-B accuracy: 15.4 -> 9.5 GMACs at 83.5% accuracy | Improved |
| replace ViT backbone with ThinkingViT in Segmenter | Segmenter · ADE20K / Cityscapes | mIoU | Improved |
Closest papers by meaning
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.88)
- MatFormer: Nested Transformer for Elastic Inference (NeurIPS 2024 · similarity 0.88)
- Elastic ViTs from Pretrained Models without Retraining (NeurIPS 2025 · similarity 0.87)
- RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers (ICML 2025 · similarity 0.87)
- Matryoshka Query Transformer for Large Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- CanViT: Toward Active-Vision Foundation Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
