Research map / Architectures and efficiency / Attention and transformers

You Only Need Less Attention at Each Stage in Vision Transformers

CVPR 2024 · Architectures and efficiency › Attention and transformers · approach: Vision transformer design · 21 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LaViT, a hierarchical ViT that computes full self-attention only in the first few layers of each stage and derives later-layer attention by linearly transforming stored attention maps, with residual attention downsampling and a diagonality-preserving loss.

Problem

Self-attention in ViTs has quadratic cost in token count, and deep ViTs suffer from attention saturation where attention maps change little across layers.

Approach

Compute vanilla attention only in the initial layers of each stage, then simulate attention in later Less-Attention layers via learned linear transforms (with a transpose) of stored attention matrices; add residual attention across stage downsampling via DWConv and 1x1 conv, and regularize with a diagonality-preserving loss.

Key results

• LaViT-S reaches 82.6% top-1 on ImageNet-1K with 3.3 GFLOPs, versus Swin-T at 81.2% with 4.5 GFLOPs.
• LaViT-B reaches 83.1% top-1 with 6.1 GFLOPs and 39.6M params, above PVT-L at 81.7%.
• Adding the Less-Attention module to ViT, DeiT, DeepViT, CeiT, HVT, PVT and Swin reduces FLOPs by roughly 9-26% while improving top-1 accuracy by 0.3-1.2 points.

Measured effects reported in the paper

Surprising findings

• Removing the diagonality-preserving loss drops LaViT-T accuracy from 79.2% to 59.1%, a far larger effect than removing the attention residual module.
• Starting Less-Attention at the second layer of a stage decreases accuracy, so the first MHSA layer must be kept before attention reuse.

Limitations

• Gains on segmentation with Semantic FPN are modest (+0.9 mIoU over PVT-S).
• Starting Less-Attention too early hurts accuracy, and later starting points raise computation, so the design needs careful layer selection.
• Comparisons exclude high-performance but computationally intensive models such as Swin-B-V2 and ConvNeXt-B by design.

Experimental setup

Datasets: ImageNet-1K COCO2017 ADE20K

Benchmarks: ImageNet-1K COCO2017 ADE20K

Metrics: Top-1 accuracy FLOPs throughput box AP mIoU cosine similarity of attention maps

Baselines: ResNet RegNet EfficientNet DeiT PVT PVTv2 DynamicViT EViT LiT PPT Swin Transformer EfficientViT

Builds on: PVT DeiT ResNet DeepViT

Closest papers by meaning

• Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice (ICLR 2022 · similarity 0.91)
• Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.91)
• RegionViT: Regional-to-Local Attention for Vision Transformers (ICLR 2022 · similarity 0.91)
• QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion (NeurIPS 2024 · similarity 0.89)
• FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.89)
• LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.89)