Research map / Architectures and efficiency / Attention and transformers
Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mobile-Attention, a kernel-based linear attention with small per-head dimension (d=4) and a head-competition mechanism based on information flow, and plugs it into DeiT, PVTv2 and EfficientFormerV2 to get mobile-friendly ViTs.
Problem
Quadratic cost of ViT self-attention limits mobile deployment; kernel-based linear attention is cheaper but reducing per-head dimension for mobile efficiency makes many small heads weak, hurting accuracy.
Approach
Reduce per-head dimension to a tiny constant (d=4) for linear-time attention, and recover capacity with a head-competition mechanism that reweights value heads by competed outgoing flow and gates results by competed incoming flow (adapted from Flowformer's information flow, applied across heads rather than tokens).
Key results
- DeiT-S-MobiAtt reaches 80.0% top-1 vs 79.8% for DeiT-S and Castling-DeiT-S, with 7.2 ms CoreML latency vs 9.0 ms for DeiT-S.
- Removing head-competition drops DeiT-S-MobiAtt from 80.0% to 76.4% top-1 at d=4.
- EfficientformerV2-L-MobiAtt reaches 46.0 semantic mIoU on ADE20K and 44.9 box AP on COCO, versus 45.2 mIoU and 44.7 box AP for EfficientFormerV2-L (mixed comparison).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard attention with Mobile-Attention | DeiT-S · ImageNet-1K val | Top-1 Acc(%): 79.8 -> 80.0 | Helped |
| Replace standard attention with Mobile-Attention | PVT-v2-b2 · ImageNet-1K val | CoreML latency: 101.0 -> 65.6 | Helped |
| Replace standard attention with Mobile-Attention | EfficientFormerV2-S2 · ImageNet-1K val | Top-1 Acc(%): 81.6 -> 82.1 | Helped |
| Replace standard attention with Mobile-Attention | EfficientFormerV2-S2 · ImageNet-1K val | CoreML latency: 1.6 -> 1.2 | Helped |
| Replace attention in the last two stages with Mobile-Attention | EfficientFormerV2-S2 · COCO 2017 val | APbox: 43.4 -> 43.7 | Helped |
| Replace attention in the last two stages with Mobile-Attention | EfficientFormerV2-S2 · ADE20K val | mIoU: 42.4 -> 43.1 | Helped |
| Add the head-competition mechanism (ablation) | DeiT-S-MobiAtt · ImageNet-1K val | Top-1 Acc (%): 76.4 -> 80.0 | Helped |
| Use vanilla design instead of Mobile-Attention head competition (ablation) | DeiT-S-MobiAtt · ImageNet-1K val | Top-1 Acc (%): 80.0 -> 79.0 | Hurt |
| Replace head competition with SE (ablation) | DeiT-S-MobiAtt · ImageNet-1K val | Top-1 Acc (%): 80.0 -> 78.3 | Hurt |
| Replace head competition with GLU (ablation) | DeiT-S-MobiAtt · ImageNet-1K val | Top-1 Acc (%): 80.0 -> 77.5 | Hurt |
| Reduce the per-head dimension (ablation) | DeiT-S-MobiAtt · ImageNet-1K | CoreML latency | Helped |
| Reduce the per-head dimension (ablation) | DeiT-S-MobiAtt · ImageNet-1K | accuracy | Hurt |
Surprising findings
- A vanilla multi-head design with more heads (no competition) performs worse than the original DeiT, showing that many small heads alone hurt capacity.
- Using SE or GLU channel attention without head competition does not preserve accuracy (78.3 and 77.5 vs 79.8 for DeiT-S).
Limitations
- Small per-head dimension degrades accuracy when d is too small (Figure 4b), so d=4 is a trade-off.
- Downstream detection and segmentation compare only backbone latency, not full pipeline latency, since some modules are not CoreML-deployable.
Experimental setup
Datasets: ImageNet-1K COCO 2017 ADE20K
Benchmarks: ImageNet-1K COCO 2017 ADE20K
Metrics: top-1 accuracy CoreML latency (ms) A100 latency (ms) Pixel 6 latency (ms) box AP mask AP mIoU GMACs params
Baselines: DeiT PVT-v2 EfficientFormerV2 Hydra-attention Castling-DeiT-S EdgeViT-XXS ConvNeXt-T Swin-T PoolFormer MobileViT-XS
Builds on: Flowformer DeiT EfficientFormerV2 PVTv2 Hydra attention
Closest papers by meaning
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.89)
- CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction (CVPR 2025 · similarity 0.88)
- SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design (CVPR 2024 · similarity 0.88)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.87)
- RepViT: Revisiting Mobile CNN From ViT Perspective (CVPR 2024 · similarity 0.87)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.