Research map / Architectures and efficiency / Attention and transformers
Skip-Attention: Improving Vision Transformers by Paying Less Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SkipAt, which skips the full MSA block in several ViT layers and replaces it with a light parametric function (FC-depthwise conv-FC-ECA) that reuses an earlier layer's MSA output, improving throughput at same or higher accuracy.
Problem
Vision transformers compute expensive self-attention in every layer, which scales quadratically with token count and limits efficiency, especially for dense prediction tasks where token sampling does not apply.
Approach
Observes high CKA correlation of CLS attention and MSA representations across adjacent layers, then skips MSA in middle layers by reusing an earlier layer's MSA output through a lightweight parametric function (two linear layers, a depth-wise convolution and ECA) instead of copying it directly.
Key results
- On ImageNet-1K, ViT-T/16 + SkipAt reaches 72.9% top-1 vs 72.8% for ViT-T with 19% higher throughput (6.9k vs 5.8k img/s); ViT-B + SkipAt reaches 82.2% vs 81.8%.
- On ADE20K with UperNet, ViT-T + SkipAt gets 40.6 mIoU vs 37.3 for ViT-T at 34.7 vs 24.1 throughput.
- On SIDD, Uformer-B + SkipAt matches Restormer-level PSNR (39.94 vs 39.89 for Uformer-B) with 77 vs 89 GFLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SkipAt to ViT-T/16 | ViT-T/16 · ImageNet-1K | top-1 accuracy (%): 72.8 -> 72.9 | Helped |
| Add SkipAt to ViT-S/16 | ViT-S/16 · ImageNet-1K | top-1 accuracy (%): 79.8 -> 80.2 | Helped |
| Add SkipAt to ViT-B/16 | ViT-B/16 · ImageNet-1K | top-1 accuracy (%): 81.8 -> 82.2 | Helped |
| Add SkipAt to ViT-T/16 | ViT-T/16 · ImageNet-1K | throughput: 5.8 -> 6.9 | Helped |
| Add SkipAt to DINO ViT-S/16 | DINO ViT-S/16 · ImageNet-1K, 100 epochs | linear-probe accuracy: 73.6% -> 74.1% | Helped |
| Add SkipAt to ViT-T | ViT-T · Pascal VOC2012 val | Jaccard: 32.2 -> 38.0 | Helped |
| Add SkipAt to ViT-S | ViT-S · Pascal VOC2012 val | Jaccard: 29.0 -> 34.0 | Helped |
| Use identity mapping instead of SkipAt parametric function (ablation) | ViT-T/16 with MSA skipped in layers 3–8 · ImageNet-1K, 100 epochs | top-1 accuracy: 65.8 -> 61.1 | Hurt |
| Replace SkipAt with a standard 5×5 convolution (ablation) | ViT-T/16 with MSA skipped in layers 3–8 · ImageNet-1K, 100 epochs | throughput: 5.2 vs 6.9 | Hurt |
| Use depth-wise convolution without the full SkipAt function (ablation) | ViT-T/16 with MSA skipped in layers 3–8 · ImageNet-1K, 100 epochs | top-1 accuracy: 65.6 vs 65.8 | No clear effect |
| Reduce SkipAt depth-wise convolution kernel from 5×5 to 3×3 (ablation) | ViT-T/16 · ImageNet-1K, 100 epochs | top-1 accuracy: 67.7 -> 67.1 | Hurt |
| Increase SkipAt depth-wise convolution kernel from 5×5 to 7×7 (ablation) | ViT-T/16 · ImageNet-1K, 100 epochs | top-1 accuracy: 67.7 -> 67.4 | Hurt |
Surprising findings
- Skipping MSA blocks in a pretrained ViT-T without retraining only drops top-1 from 72.8% to 47.8%, suggesting strong cross-layer redundancy.
- Simply copying (identity) the earlier MSA output reduces accuracy by 4.7 points, so the parametric function is needed to recover performance.
Limitations
- Parametric function requires retraining from scratch; skipping in pretrained models without retraining causes large accuracy drops.
- Video denoising experiments use a simpler identity-based attention reuse rather than the proposed parametric function.
- Unsupervised segmentation on Pascal VOC is comparable to or below DINO baseline for SkipAt (44.7 vs 45.3 Jaccard).
Experimental setup
Datasets: ImageNet-1K ADE20K SIDD DAVIS Pascal VOC2012
Benchmarks: ImageNet-1K Pascal VOC2012 ADE20K SIDD DAVIS
Metrics: top-1 accuracy throughput GFLOPs parameters linear probing accuracy mIoU PSNR SSIM Jaccard similarity CorLoc
Baselines: A-ViT Dynamic ViT SViTE SPViT ATS PS-ViT HVT Rev-ViT Swin-T ConvNeXt-T Restormer UniFormer
Builds on: Vision Transformer ResNeXt Efficient Channel Attention DINO Uformer
Closest papers by meaning
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.91)
- AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (CVPR 2022 · similarity 0.90)
- Cutting the Skip: Training Residual-Free Transformers (ICLR 2026 · similarity 0.89)
- MLP Can Be A Good Transformer Learner (CVPR 2024 · similarity 0.89)
- LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.88)
- PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.