Research map / Architectures and efficiency / Neural architecture search
GLiT: Neural Architecture Search for Global and Local Image Transformer
Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, Wanli Ouyang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace 1 of 3 self-attention heads with a Conv1d head (ablation) | DeiT-Tiny · ImageNet | top-1 accuracy (%): 72.20 -> 72.89 | Improved |
| replace 2 of 3 self-attention heads with Conv1d heads (ablation) | DeiT-Tiny · ImageNet | top-1 accuracy (%): 72.20 -> 73.98 | Improved |
| replace all self-attention heads with Conv1d heads (ablation) | DeiT-Tiny · ImageNet | top-1 accuracy (%): 72.20 -> 71.02 | Hurt |
| search global-local distribution per block (ablation) | DeiT-Tiny · ImageNet | top-1 accuracy (%): 72.2 -> 75.4 | Improved |
| search the detailed sub-module architecture on top of distribution (ablation) | GLiT-Tiny · ImageNet | top-1 accuracy (%): 75.4 -> 76.3 | Improved |
| searched distribution vs best uniform human-designed global-local mix (ablation) | GLiT-Tiny · ImageNet | top-1 accuracy (%): 73.98 -> 75.4 | Improved |
| use GLiT search space instead of NLP-NAS search space (ablation) | GLiT-Tiny · ImageNet | top-1 accuracy (%): 73.4 -> 76.3 | Improved |
| use hierarchical search instead of SPOS (ablation) | GLiT-Tiny · ImageNet | top-1 accuracy (%): 72.7 -> 76.3 | Improved |
| use hierarchical search instead of random search (ablation) | GLiT-Tiny · ImageNet | top-1 accuracy (%): 73.3 -> 76.3 | Improved |
| use Conv1D rather than local-area self-attention for locality (ablation) | GLiT-Tiny · ImageNet | top-1 accuracy (%): 72.4 -> 76.3 | Improved |
| use Conv2D instead of Conv1D (ablation) | GLiT-Tiny · ImageNet | top-1 accuracy (%): 76.3 -> 76.4 | No clear effect |
| GLiT-Tiny vs DeiT-Tiny under similar FLOPs | DeiT-Tiny · ImageNet | top-1 accuracy (%): 72.2 -> 76.3 | Improved |
Closest papers by meaning
- Masked Distillation Advances Self-Supervised Transformer Architecture Search (ICLR 2024 · similarity 0.89)
- AutoFormer: Searching Transformers for Visual Recognition (ICCV 2021 · similarity 0.88)
- BossNAS: Exploring Hybrid CNN-Transformers With Block-Wisely Self-Supervised Neural Architecture Search (ICCV 2021 · similarity 0.88)
- Searching the Search Space of Vision Transformer (NeurIPS 2021 · similarity 0.88)
- L-SWAG: Layer-Sample Wise Activation with Gradients Information for Zero-Shot NAS on Vision Transformers (CVPR 2025 · similarity 0.86)
- Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
