Research map / Architectures and efficiency / Attention and transformers
Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vision-RWKV adapts the RWKV linear-attention architecture to vision with bidirectional global attention (Bi-WKV), a quad-directional token shift (Q-Shift), and scale-up stability tweaks, matching or beating ViT on classification, detection and segmentation at lower cost on high-resolution inputs.
Problem
Vision Transformers have quadratic attention cost that limits high-resolution and long-sequence vision processing, and prior linear-attention vision models (e.g., Vim, VMamba) were only validated at small scale and suffered training instability when scaled up.
Approach
Converts RWKV's causal attention into bidirectional global attention computed in linear time via a forward/backward RNN form, adds a relative positional bias normalized by token count and bounded exponentials, and introduces Q-Shift, a four-direction token shift for 2D images, plus extra layer norms and layer scale for stable scaling.
Key results
- VRWKV-T reaches 75.1% top-1 on ImageNet-1K trained only on IN-1K, 2.9 points above DeiT-T with slightly fewer FLOPs (1.2G vs 1.3G).
- VRWKV-L reaches 86.0% top-1 at 384x384 after ImageNet-22K pre-training, above ViT-L (85.2 reported in the table), and 50.6 box AP on COCO with Mask R-CNN and ViT-Adapter, versus 48.7 for ViT-L with window attention.
- At 2048x2048 (16384 tokens), VRWKV-T inference is reported as 10x faster than ViT-T with 80% lower memory consumption on an A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Q-Shift instead of no token shift (ablation) | VRWKV-T · ImageNet-1K | top-1 accuracy: 71.5 -> 75.1 | Helped |
| replace original RWKV token shift with Q-Shift (ablation) | VRWKV-T · ImageNet-1K | top-1 accuracy: 74.4 -> 75.1 | Helped |
| add bidirectional attention instead of causal attention (ablation) | VRWKV-T · ImageNet-1K | top-1 accuracy: 72.8 -> 75.1 | Helped |
| add MAE pre-training (ablation) | VRWKV-L · ImageNet-1K val, 384² | top-1 accuracy: 86.0 -> 86.2 | Helped |
| pre-train on Bamboo-47K (ablation) | VRWKV-L · ImageNet-1K val, 384² | top-1 accuracy: 86.0 -> 86.5 | Helped |
| replace DeiT-T with VRWKV-T | ImageNet-1K classification · ImageNet-1K val, 224² | top-1 accuracy: 72.2 -> 75.1 | Helped |
| replace ViT-L with VRWKV-L | ImageNet-1K classification · ImageNet-1K val, 384² | top-1 accuracy: 85.2 -> 86.0 | Helped |
| use VRWKV-T instead of ViT-T with window attention | COCO detection backbone · COCO val2017 | box AP: 41.1 -> 41.7 | Helped |
| use VRWKV-S instead of ViT-S with global attention | COCO detection backbone · COCO val2017 | box AP: 44.9 -> 44.8 | No clear effect |
| use VRWKV-L instead of ViT-L | COCO detection backbone · COCO val2017 | box AP: 48.7 -> 50.6 | Helped |
| use VRWKV-S instead of ViT-S | ADE20K segmentation backbone · ADE20K val | mIoU: 46.2 -> 47.2 | Helped |
| use VRWKV-L instead of ViT-L | ADE20K segmentation backbone · ADE20K val | mIoU: 53.4 -> 53.5 | No clear effect |
Surprising findings
- The Q-Shift operation implemented in plain PyTorch slows the model overall, despite reducing FLOPs; the authors attribute this to implementation rather than design.
- Vim reportedly achieves higher ImageNet top-1 than VRWKV at tiny and small scales (76.1 vs 75.1; 80.5 vs 80.1).
Limitations
- Q-Shift and Bi-WKV lack hardware-level CUDA optimization, so wall-clock gains are smaller than FLOP reductions and flash attention is faster at the kernel level.
- Vim slightly outperforms VRWKV on tiny and small ImageNet classification.
- Training relies on large-scale pre-training (ImageNet-22K) to reach top results for the large model.
Experimental setup
Datasets: ImageNet-1K ImageNet-22K Bamboo-47K COCO ADE20K
Benchmarks: ImageNet-1K COCO val2017 ADE20K
Metrics: top-1 accuracy box AP mask AP mIoU FLOPs inference speed GPU memory
Baselines: ViT DeiT Swin PVT XCiT ResNet Vision Mamba
Builds on: RWKV Vision Transformer ViT-Adapter MAE
Closest papers by meaning
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.86)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.86)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.86)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.86)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.86)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.