Research map / Architectures and efficiency / Attention and transformers
Breaking the Low-Rank Dilemma of Linear Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rank-Augmented Linear Attention (RALA), which reweights KV-buffer tokens and applies token-specific modulation to the output to raise feature rank, and builds the RAVLT vision backbone that matches Softmax-attention models on ImageNet, detection and segmentation at linear cost.
Problem
Linear attention is far cheaper than Softmax attention but loses substantial accuracy in vision; the paper attributes this to low-rank output features that limit spatial modeling.
Approach
Analyze rank at two stages of linear attention: weight each token's contribution to the KV buffer by a context-aware coefficient derived from the global query (raising KV buffer rank), then modulate each output token channel-wise by a transformed copy of its input (restoring full output rank).
Key results
- RAVLT-S reaches 84.4% Top-1 on ImageNet-1K with 26M params and 4.6G FLOPs, and RAVLT-B reaches 85.5% with 48M params and 9.9G FLOPs.
- On the DeiT-T setting, RALA reaches 75.1% Top-1 versus 72.2% for DeiT-T Softmax and 74.1% for Focused Linear Attention.
- RAVLT-B reaches 51.9 mIoU with Semantic FPN on ADE20K, surpassing RMT-L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add KV buffer augmentation (ablation) | RAVLT-T · ImageNet-1K | Top-1 accuracy: 82.1(-0.7) | Helped |
| add output-feature augmentation (ablation) | RAVLT-T · ImageNet-1K | Top-1 accuracy: 82.5(-0.3) | Helped |
| add conditional positional encoding (CPE) (ablation) | RAVLT-T · ImageNet-1K | Top-1 accuracy: 82.7(-0.1) | Helped |
| replace the projection in ϕ with identity (ablation) | RAVLT-T · ImageNet-1K | Top-1 accuracy: 82.8(-0.0) | No clear effect |
| replace the projection in ϕ with tanh (ablation) | RAVLT-T · ImageNet-1K | Top-1 accuracy: 82.7(-0.1) | No clear effect |
| replace ELU + 1 in κ with ReLU (ablation) | RAVLT-T · ImageNet-1K | Top-1 accuracy: 82.8(-0.0) | No clear effect |
| replace ELU + 1 in κ with Softmax (ablation) | RAVLT-T · ImageNet-1K | Top-1 accuracy: 82.7(-0.1) | No clear effect |
| use RALA instead of other linear attention mechanisms | DeiT-T · ImageNet-1K | Top-1 accuracy: 75.1 vs. 74.1 | Helped |
Surprising findings
- Replacing Softmax attention with RALA in DeiT-T, Swin-T and Swin-S improves accuracy over the Softmax baseline, not only over other linear attentions.
- Changing the feature map phi or kernel kappa (linear projection, identity, tanh, ReLU, Softmax) has little effect on accuracy, which the authors attribute to the rank augmentation rather than the specific functions.
Limitations
- Rank is measured on single heads of a small setting (N=196, d=64), so the rank-performance link is shown empirically rather than proven.
- Comparisons to some Softmax baselines use differing training recipes and parameter counts, complicating fair comparison.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K
Benchmarks: ImageNet-1K ImageNet-V2 COCO ADE20K
Metrics: Top-1 accuracy AP box AP mask mIoU params FLOPs throughput matrix rank
Baselines: DeiT Swin Transformer RMT MLLA FLatten Transformer SOFT BiFormer MogaNet InternImage Enhanced Linear Attention Efficient Attention Focused Linear Attention
Builds on: linear attention FLatten Transformer Enhanced Linear Attention Efficient Attention conditional positional encoding DeiT
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.92)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.88)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.87)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
- Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers (ICML 2024 · similarity 0.87)
- LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.