Research map / Architectures and efficiency / Attention and transformers
Fast Vision Transformers with HiLo Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MSA with HiLo (ablation) | LITv2-S · ADE20K | mIoU: 43.7 -> 44.3 | Helped |
| Replace MSA with HiLo (ablation) | LITv2-S · ImageNet-1K | Top-1 accuracy: 82.3 -> 82.0 | Hurt |
| Add a 3×3 depthwise convolution to each FFN (ablation) | LITv1-S · ImageNet-1K | Top-1 accuracy: 81.5 -> 82.5 | Helped |
| Add a 3×3 depthwise convolution to each FFN (ablation) | LITv1-S · COCO RetinaNet | AP: 41.6 -> 45.1 | Helped |
| Remove relative positional encoding (ablation) | LITv1-S with ConvFFN · COCO RetinaNet | FPS: 3.1 -> 13.3 | Helped |
| Remove relative positional encoding (ablation) | LITv1-S with ConvFFN · COCO RetinaNet | AP: 45.1 -> 44.7 | Hurt |
| Compute Lo-Fi queries from pooled features rather than the original input (ablation) | LITv2-S · ImageNet-1K | throughput: 1,471 -> 1,084 | Hurt |
| Use window size 4 instead of 2 (ablation) | LITv2-S · CIFAR-100 | Top-1 accuracy: 85.1 -> 84.4 | Hurt |
| Set α=0, retaining only Hi-Fi (ablation) | HiLo · ImageNet-1K | performance | Hurt |
| Use α=1.0, retaining only Lo-Fi (ablation) | HiLo · ADE20K semantic segmentation | mIoU: 43.7% | Hurt |
| Increase HiLo window size from 2 to 4 (ablation) | LITv2 · COCO detection and instance segmentation | FPS: 18.7 -> 20.4 | Helped |
| Increase HiLo window size from 2 to 4 (ablation) | LITv2-S · COCO RetinaNet | AP: 44.0 -> 43.7 | Hurt |
Closest papers by meaning
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
- EfficientViT: Memory Efficient Vision Transformer With Cascaded Group Attention (CVPR 2023 · similarity 0.87)
- Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice (ICLR 2022 · similarity 0.87)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.87)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.86)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.