Research map / Architectures and efficiency / Attention and transformers
AttnZero: Efficient Attention Discovery for Vision Transformers
Lujun Li, Zimian Wei, Peijie Dong, Wenhan Luo, Wei Xue, Qifeng Liu, Yike Guo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax attention with AttnZero-discovered linear attention | DeiT-T · ImageNet-1k (224x224) | Top-1 accuracy (%): 72.2 -> 74.9 | Improved |
| replace self-attention with AttnZero in larger or stronger backbones | Swin-S, Swin-B, CSwin-T · ImageNet-1k (224x224) | Top-1 accuracy (%): 83.0 -> 83.2 (Swin-S); 83.5 -> 83.7 (Swin-B); 82.7 -> 82.9 (CSwin-T) | Improved |
| use AttnZero search instead of random search in the same search space (ablation) | DeiT-T · CIFAR-100 | Top-1 accuracy (%): 76.04 -> 77.68 | Improved |
| use multi-objective search over several ViTs instead of single-objective search (ablation) | AttnZero search · ImageNet-1k | Top-1 accuracy (%): +0.9% | Improved |
| search attention on one ViT (Swin) only (ablation) | AttnZero search · DeiT, PVT transfer | Top-1 accuracy | Hurt |
| add program checking and rejection protocols to the search (ablation) | AttnZero search · search efficiency | search speedup: more than 5x search acceleration | Improved |
| use advanced normalization and activation operators (vs new binary operators) (ablation) | search space · Attn-Bench-101 (AutoFormer, CIFAR-100) | Top-1 accuracy | Improved |
| use computation graphs i3n302 and i3n303 (ablation) | search space · Attn-Bench-101 (AutoFormer, CIFAR-100) | Top-1 accuracy | Improved |
| use other linear attentions (Hydra, Efficient Attn, Linear Angular) in place of softmax | DeiT-T · ImageNet-1k (224x224) | Top-1 accuracy (%): 72.2 -> 68.3 (Hydra) | Hurt |
| replace attention with AttnZero in PVT-T for detection | PVT-T Mask R-CNN · COCO (Mask R-CNN 1x) | box AP: 36.7 -> 40.0 | Improved |
| replace attention with AttnZero in PVT-T for segmentation | PVT-T SemanticFPN · ADE20K | mIoU: 36.57 -> 39.16 | Improved |
Closest papers by meaning
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.86)
- PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer (ICLR 2025 · similarity 0.86)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.86)
- Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers (ICML 2024 · similarity 0.84)
- Learning Endogenous Attention for Incremental Object Detection (CVPR 2025 · similarity 0.84)
- CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
