Research map / Architectures and efficiency / Attention and transformers
Intriguing Properties of Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| ViT (DeiT-S) instead of CNN (ResNet50) | image classifier · ImageNet 50% Random PatchDrop | top-1: 0.1% -> 70% | Helped |
| ViT instead of CNN | image classifier · ImageNet-C | mCE: 65.3 -> 54.6 | Helped |
| train on Stylized ImageNet (no augmentation) | DeiT-S · ImageNet-C | mCE: 54.6 -> 84.0 | Hurt |
| add shape-token distillation from ResNet50-SIN (ablation) | DeiT-S-SIN · ImageNet val | top-1: 52.5 -> 75.3 | Helped |
| add shape-token distillation from ResNet50-SIN (ablation) | DeiT-S-SIN · shape bias | shape bias: 0.93 -> 0.47 | Hurt |
| shape distillation (ablation) | DeiT-S · PASCAL-VOC12 | Jaccard index: 29.2 -> 42.4 | Helped |
| concatenate class tokens of blocks 9-12 vs last block only (ablation) | DeiT-S off-the-shelf · CUB | top-1: 68.16 -> 73.58 | Helped |
| concatenate class tokens of blocks 9-12 vs last block only (ablation) | DeiT-S off-the-shelf · iNaturalist | top-1: 38.28 -> 45.15 | Helped |
| add averaged patch tokens to class tokens (blocks 9-12) (ablation) | DeiT-S off-the-shelf · CUB | top-1: 73.58 -> 73.37 | No clear effect |
| use class tokens from all blocks vs blocks 9-12 (ablation) | DeiT-S off-the-shelf · Flowers | top-1: 90.00 -> 91.38 | Helped |
| remove positional encoding (ablation) | DeiT · ImageNet shuffled patches | top-1 | Mixed |
| reduce patch size in training (ablation) | DeiT-T · ImageNet shuffled patches | top-1 | Hurt |
Closest papers by meaning
- Vision Transformers Are Robust Learners (AAAI 2022 · similarity 0.93)
- Vision Transformers provably learn spatial structure (NeurIPS 2022 · similarity 0.92)
- Towards Robust Vision Transformers: Path Dependency Analysis and a Simple Two-Stage Adversarial Training (CVPR 2026 · similarity 0.92)
- Interpreting vision transformers via residual replacement model (NeurIPS 2025 · similarity 0.90)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.90)
- Your ViT is Secretly an Image Segmentation Model (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.