AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Tokens-to-Token ViT: Training Vision Transformers From Scratch on ImageNet

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, Shuicheng Yan

ICCV 2021 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 2,606 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add T2T module (progressive tokenization) (ablation)T2T-ViT-14 backbone · ImageNet-1k val (from scratch)top-1 accuracy (%): 79.5 -> 81.5Improved
use Transformer layer instead of Performer in T2T module (ablation)T2T-ViT-14 · ImageNet-1k val (from scratch)top-1 accuracy (%): 81.5 -> 81.7Improved
use T2T soft-split module instead of 3 convolution layers (ablation)T2T-ViT-14 · ImageNet-1k val (from scratch)top-1 accuracy (%): 80.8 -> 81.5Improved
use deep-narrow instead of shallow-wide backbone (ablation)T2T-ViT-14 · ImageNet-1k val (from scratch)top-1 accuracy (%): 78.8 -> 81.5Improved
use deep-narrow (16 layers, 384 dim) instead of baseline (ablation)ViT-S/16 · ImageNet-1k val (from scratch)top-1 accuracy (%): 78.1 -> 79.0Improved
use shallow-wide (4 layers, 1024 dim) structure (ablation)ViT-S/16 · ImageNet-1k val (from scratch)top-1 accuracy (%): 78.1 -> 69.9Hurt
add dense connection (ablation)ViT-S/16 · ImageNet-1k val (from scratch)top-1 accuracy (%): 78.1 -> 76.8Hurt
add dense connection (ablation)T2T-ViT-14 · ImageNet-1k val (from scratch)top-1 accuracy (%): 81.5 -> 80.6Hurt
add SE channel attention (ablation)ViT-S/16 · ImageNet-1k val (from scratch)top-1 accuracy (%): 78.1 -> 78.4Improved
add SE channel attention (ablation)T2T-ViT-14 · ImageNet-1k val (from scratch)top-1 accuracy (%): 81.5 -> 81.6No clear effect
use 32 heads (ResNeXt structure) (ablation)T2T-ViT-14 · ImageNet-1k val (from scratch)top-1 accuracy (%): 81.5 -> 81.5No clear effect
add Ghost operations (ablation)T2T-ViT-14 · ImageNet-1k val (from scratch)top-1 accuracy (%): 81.5 -> 79.5Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy