Research map / Detection and segmentation / CNN features

FasterViT: Fast Vision Transformers with Hierarchical Attention

ICLR 2024 · Detection and segmentation › CNN features · approach: Plain and high-resolution ViTs · 35 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FasterViT, a hybrid CNN-ViT family whose Hierarchical Attention (carrier tokens summarizing local windows plus attention among them) gives a high ImageNet accuracy-throughput Pareto front and competitive dense-prediction results.

Problem

Vision transformers with global self-attention are quadratic in cost and slow on high-resolution inputs, while window-based attention lacks cross-window communication; the goal is a fast, high-throughput ViT with efficient global context.

Approach

Hierarchical Attention (HAT) learns carrier tokens that summarize each local window; carrier tokens attend globally among themselves and then jointly with their own window's tokens, giving cross-window and long-range interaction at near-linear cost. Combined with a conv-based early-stage design optimized for GPU throughput.

Key results

• FasterViT-2 reaches 84.2% ImageNet-1K top-1 at 3161 img/s on A100 (batch 128), outperforming ConvNeXt-T by 2.2% at comparable throughput.
• FasterViT-4 achieves 52.9 box AP and 45.8 mask AP on COCO with Cascade Mask R-CNN, running 15% faster than ConvNeXt-B and 30% faster than Swin-B.
• FasterViT-4 pre-trained on ImageNet-21K reaches 87.5% top-1 at 384px fine-tuning, and HAT added to Swin-T improves ImageNet top-1 from 81.3 to 81.7.

Measured effects reported in the paper

Surprising findings

• Removing the HAT block after training costs -1.49% top-1, while a window size of 14x14 without carrier tokens gives +0.1% accuracy when trained from scratch but costs about 10% throughput.
• Layer normalization is critical in transformer stages: replacing it with batch normalization drops accuracy by 0.7%.

Limitations

• Throughput advantages are measured on specific hardware and frameworks (A100, PyTorch, TensorRT), and the paper notes EfficientNetV2 beats it on A6000.
• Large variants rely on ImageNet-21K pretraining and very large parameter counts (up to 1.36B), limiting parameter efficiency.
• Gains over strong baselines on dense tasks are modest (for example, +0.2 box AP over ConvNeXt-B).

Experimental setup

Datasets: ImageNet-1K ImageNet-21K MS COCO ADE20K ImageNet-A ImageNet-R ImageNetV2

Benchmarks: ImageNet-1K MS COCO ADE20K ImageNet-A ImageNet-R ImageNetV2

Metrics: top-1 accuracy throughput (img/sec) box AP mask AP mIoU FLOPs

Baselines: ConvNeXt Swin Transformer SwinV2 Twins EdgeViT EfficientFormer MaxViT DeiT Cascade Mask R-CNN UPerNet DINO

Builds on: Swin Transformer SwinV2 Twins EdgeViT ConvNeXt

Closest papers by meaning

• EfficientViT: Memory Efficient Vision Transformer With Cascaded Group Attention (CVPR 2023 · similarity 0.90)
• You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.89)
• SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design (CVPR 2024 · similarity 0.88)
• QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion (NeurIPS 2024 · similarity 0.88)
• RepViT: Revisiting Mobile CNN From ViT Perspective (CVPR 2024 · similarity 0.88)
• Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)