Research map / Architectures and efficiency / Attention and transformers
CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CARE, a decoupled dual-interactive linear attention for mobile-friendly visual Transformers that splits channels asymmetrically between local convolution and linear attention, plus a dynamic memory unit and dual interaction module.
Problem
Linear-complexity visual Transformers either give small efficiency gains or suffer accuracy drops, making them hard to deploy on resource-constrained mobile devices; stacking local convolutions and linear attention is inflexible and costly.
Approach
Asymmetrically split channels so that a local depth-wise convolution branch and a linear attention branch learn separately (d1 < d2), then fuse them with a dynamic memory unit carrying features across layers and a dual interaction module (local-global and cross-layer), reducing the quadratic channel overhead of linear attention.
Key results
- CARE-S0 reaches 78.4% top-1 on ImageNet-1K at 0.7 GMACs; CARE-S2 reaches 82.1% at 1.9 GMACs.
- CARE-S2 costs about half the GMACs of MLLA-T and runs about 2.5x faster on iPhone13 and iPad Pro, with accuracy 1.4% lower than MLLA-T (83.5%).
- Ablation on CARE-S2: asymmetric decoupling (82.1%, 1.9 GMACs) matches symmetric (82.1%, 2.0 GMACs) with lower cost; stacked design gives 81.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CARE-S0 instead of MobileViG-T at 0.7 GMACs | Image classification models · ImageNet-1K | top-1 accuracy: 75.7 -> 78.4 | Helped |
| Use CARE-S2 instead of EdgeViT-S at 1.9 GMACs | Image classification models · ImageNet-1K | top-1 accuracy: 81.0 -> 82.1 | Helped |
| Use CARE-S2 instead of MLLA-T | Image classification models · ImageNet-1K | GMACs: 4.2 -> 1.9 | Helped |
| Use CARE-S2 instead of MLLA-T | Mask R-CNN backbones · COCO val | box AP: 46.8 -> 43.1 | Hurt |
| Use asymmetrical rather than symmetrical feature decoupling (ablation) | CARE-S2 · ImageNet-1K | GMACs: 2.0 -> 1.9 | Helped |
| Replace asymmetrical decoupling with stacked local enhancement (ablation) | CARE-S2 · ImageNet-1K | top-1 accuracy: 82.1 -> 81.4 | Hurt |
| Remove local inductive bias (ablation) | CARE-S2 · ImageNet-1K | top-1 accuracy: 82.1 -> 77.3 | Hurt |
| Add the first interaction block (ablation) | CARE-S0 · ImageNet-1K | top-1 accuracy: 76.2 -> 78.4 | Helped |
| Add the second interaction block (ablation) | CARE-S0 · ImageNet-1K | top-1 accuracy: 70.9 -> 78.4 | Helped |
| Add memory information to Inter2 (ablation) | CARE-S0 · ImageNet-1K | top-1 accuracy: 76.5 -> 78.4 | Helped |
| Use early-stage 1×11 and 11×1 convolutions instead of linear attention (ablation) | CARE Transformers · ImageNet-1K | top-1 accuracy: 78.3 -> 78.4; 79.9 -> 80.1; 82.0 -> 82.1 | Helped |
| Use multiscale local inductive bias (ablation) | CARE Transformers · ImageNet-1K | top-1 accuracy: 78.1 -> 78.4; 79.8 -> 80.1; 81.8 -> 82.1 | Helped |
Surprising findings
- Replacing the early 1x11 and 11x1 convolutions with linear attention in the first two stages slightly reduces accuracy.
Limitations
- No neural architecture search was used to optimize the CARE architecture.
- Method was not applied to larger models due to limited GPU resources.
- Shallow-stage linear attention reduces accuracy slightly, attributed to noisy features in early layers.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K
Benchmarks: ImageNet-1K COCO ADE20K
Metrics: top-1 accuracy GMACs latency (ms) params (M) box AP mask AP mIoU
Baselines: MLLA-T FastViT-T8 MobileViG-T MobileViGv2-T EdgeViT-S MobileOne-S3 SLAB-Swin-T FLatten-Swin-T Agent-Swin-T Swin-T
Builds on: MLLA EfficientAtt FLatten Swin Transformer
Closest papers by meaning
- Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers (ICML 2024 · similarity 0.88)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.87)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.87)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.86)
- DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images (NeurIPS 2026 · similarity 0.86)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.