Research map / Architectures and efficiency / Attention and transformers
Going Deeper With Image Transformers
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, Hervé Jégou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LayerScale (per-channel residual scaling) to a 36-block model (ablation) | DeiT-S depth 36 · ImageNet-1k val | top-1 accuracy: 81.9 -> 82.9 | Improved |
| use per-channel LayerScale instead of ReZero scalar weighting (ablation) | DeiT-S depth 36 · ImageNet-1k val | top-1 accuracy: 81.6 -> 82.9 | Improved |
| initialize LayerScale at small eps instead of 0 (ablation) | DeiT-S depth 36 · ImageNet-1k val | top-1 accuracy: 82.7 -> 82.9 | Improved |
| initialize LayerScale at constant eps instead of uniform random U[0,2eps] (ablation) | DeiT-S depth 36 · ImageNet-1k val | top-1 accuracy: 82.6 -> 82.9 | Improved |
| keep scale factors learnable instead of fixed to values from a previous LayerScale run (ablation) | DeiT-S depth 36 · ImageNet-1k val | top-1 accuracy: 81.6 -> 82.9 | Improved |
| adapt stochastic depth drop rate when adding LayerScale (ablation) | DeiT-S depth 36 · ImageNet-1k val | top-1 accuracy: 80.5 -> 83.0 | Improved |
| add class-attention layers (12 SA + 2 CA) to DeiT-S (ablation) | DeiT-S · ImageNet-1k val | top-1 accuracy: 79.9 -> 80.8 | Improved |
| insert the class token late, at layer 10 instead of layer 0 (ablation) | DeiT-S · ImageNet-1k val | top-1 accuracy: 79.9 -> 80.5 | Improved |
| add class-attention architecture on top of LayerScale (ablation) | CaiT-S36 · ImageNet-1k val | top-1 accuracy: 83.0 -> 83.2 | Improved |
| exclude the class embedding from keys of class-attention (ablation) | CaiT-S36 · ImageNet-1k val | top-1 accuracy: 83.44 -> 83.31 | No clear effect |
| remove LayerScale in the class-attention blocks (ablation) | CaiT-S36 · ImageNet-1k val | top-1 accuracy: 83.36 -> 83.44 | No clear effect |
| add a distillation token to hard distillation (ablation) | CaiT · ImageNet-1k val | top-1 accuracy: 78.4 -> 78.5 (XXS-24); 84.8 -> 84.7 (M-24) | No clear effect |
Closest papers by meaning
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ICLR 2021 · similarity 0.91)
- CMT: Convolutional Neural Networks Meet Vision Transformers (CVPR 2022 · similarity 0.89)
- Three Things Everyone Should Know about Vision Transformers (ECCV 2022 · similarity 0.89)
- Visual Transformers: Where Do Transformers Really Belong in Vision Models? (ICCV 2021 · similarity 0.89)
- A Survey on Vision Transformer (TPAMI 2022 · similarity 0.88)
- Training data-efficient image transformers & distillation through attention (ICML 2021 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
