Research map / Architectures and efficiency / Attention and transformers
RMT: Retentive Networks Meet Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RMT, a vision backbone that extends RetNet's temporal decay to a 2D Manhattan-distance spatial decay inside self-attention (MaSA), with a decomposed form giving linear-complexity global modeling.
Problem
Vision Transformer self-attention lacks explicit spatial priors and has quadratic cost in the number of tokens, limiting ViT applicability for general vision tasks.
Approach
Transforms RetNet's unidirectional 1D temporal decay into a bidirectional 2D spatial decay gamma^(|x_n-x_m|+|y_n-y_m|) inside softmax attention (Manhattan Self-Attention), and decomposes attention along image axes so the decay matrix is preserved while cost becomes linear.
Key results
- RMT-S reaches 84.1% top-1 on ImageNet-1K at 4.5 GFLOPs without extra data; RMT-L* reaches 86.1% at 18.2 GFLOPs with token labeling.
- RMT-L achieves 51.6 box AP and 45.9 mask AP with Mask R-CNN 1x on COCO; RMT-B reaches 54.5 box AP with Cascade Mask R-CNN 3x+MS.
- RMT-L reaches 52.8 mIoU on ADE20K with UperNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RMT instead of BiFormer-S | ImageNet-1K classification · ImageNet-1K | Top-1 accuracy: 83.8 -> 84.1 | Helped |
| Use RMT-S* instead of BiFormer-S* | ImageNet-1K classification with token labeling · ImageNet-1K | Top-1 accuracy: 84.3 -> 84.8 | Helped |
| Use RMT instead of InternImage-T | COCO Mask R-CNN 1× · COCO val2017 | box AP: 47.2 -> 49.0 | Helped |
| Use RMT instead of InternImage-B | COCO Mask R-CNN 1× · COCO val2017 | box AP: 48.8 -> 51.6 | Helped |
| Use RMT-B instead of CSWin-S | COCO Cascade Mask R-CNN 3×+MS · COCO val2017 | box AP: 53.7 -> 54.5 | Helped |
| Use RMT-S instead of Shunted-S | ADE20K Semantic FPN · ADE20K | mIoU: 48.2 -> 49.4 | Helped |
| Replace MaSA with vanilla attention (ablation) | RMT-T · ImageNet-1K | Top-1 accuracy: 82.4 -> 81.6(-0.8) | Hurt |
| Replace MaSA with vanilla attention (ablation) | RMT-T · COCO val2017, Mask R-CNN 1× | box AP: 47.1 -> 44.6(-2.5) | Hurt |
| Replace MaSA with vanilla attention (ablation) | RMT-T · ADE20K Semantic FPN | mIoU: 46.4 -> 43.9(-2.5) | Hurt |
| Add LCE (ablation) | RMT-T · ImageNet-1K | Top-1 accuracy: 82.1 -> 82.4 | Helped |
| Add CPE (ablation) | RMT-T · ImageNet-1K | Top-1 accuracy: 82.2 -> 82.4 | Helped |
| Add convolutional stem (ablation) | RMT-T · ImageNet-1K | Top-1 accuracy: 82.2 -> 82.4 | Helped |
Surprising findings
- Replacing the RetNet retention gating function with softmax did not improve vision results, so the authors kept softmax attention.
- Decomposed MaSA matches full MaSA on classification (84.1% both) while reducing segmentation FLOPs substantially (180G vs 246G).
Limitations
- The last stage uses the original quadratic MaSA, so linear complexity holds only for the first three stages.
- Gains over strong baselines in some settings are modest (e.g., 0.5% with token labeling vs BiFormer-S).
Experimental setup
Datasets: ImageNet-1K COCO 2017 ADE20K ImageNet-A ImageNet-R ImageNet-V2
Benchmarks: ImageNet-1K COCO 2017 ADE20K ImageNet-A ImageNet-R ImageNet-V2
Metrics: top-1 accuracy box AP mask AP mIoU FLOPs throughput
Baselines: Swin Transformer BiFormer SMT MaxViT InternImage CSWin iFormer DeiT FAT MOAT
Builds on: RetNet ALiBi Swin Transformer CPE BiFormer
Closest papers by meaning
- Learning Spatial Decay for Vision Transformers (AAAI 2026 · similarity 0.89)
- ReMoE: Region-Mixture Experts for Adversarially-Robust Vision Transformers (CVPR 2026 · similarity 0.88)
- A-ViT: Adaptive Tokens for Efficient Vision Transformer (CVPR 2022 · similarity 0.88)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.87)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.