Research map / Detection and segmentation / CNN features
DefMamba: Deformable Visual State Space Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DefMamba, a visual state space backbone whose deformable Mamba blocks learn input-dependent reference-point shifts and token scan orders, evaluated on ImageNet classification, COCO detection/segmentation, and ADE20K segmentation.
Problem
Visual Mamba methods flatten 2D images into 1D sequences with fixed scan orders, which breaks spatial adjacency and loses structural information.
Approach
A deformable state space model with a deformable scanning strategy: a learned offset network shifts sampling reference points toward informative regions and reorders tokens by predicted token-index offsets, added as a branch alongside standard forward/backward scans.
Key results
- DefMamba-T reaches 78.6% Top-1 on ImageNet-1K, above PlainMamba-L1 (77.9%) and ViM-T (76.1%); DefMamba-B reaches 84.2%, matching GrootV-T's 84.2% and exceeding VMamba-S (83.6%).
- DefMamba-S reaches 47.5 box AP and 42.8 mask AP on COCO with Mask R-CNN 1x, slightly above VMamba-T's 47.4 box AP.
- DefMamba-S reaches 48.8 mIoU (single-scale) on ADE20K, above GrootV-T (48.5) and VMamba-T (47.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DefMamba-T instead of PlainMamba-L1 | Image classification models · ImageNet-1K val | Top-1 accuracy: 77.9 -> 78.6 | Helped |
| Use DefMamba-S instead of GrootV-T | Image classification models · ImageNet-1K val | Top-1 accuracy: 83.4 -> 83.5 | Helped |
| Use DefMamba-S instead of VMamba-T | Mask R-CNN · MSCOCO 2017 val, 1× schedule | APb: 47.4 -> 47.5 | Helped |
| Use DefMamba-S instead of VMamba-T | Mask R-CNN · MSCOCO 2017 val, 1× schedule | APm: 42.7 -> 42.8 | Helped |
| Use DefMamba-S instead of GrootV-T | Semantic segmentation · ADE20K val, single-scale | mIoU: 48.5 -> 48.8 | Helped |
| Use DefMamba-S instead of GrootV-T | Semantic segmentation · ADE20K val, multi-scale | mIoU: 49.4 -> 49.6 | Helped |
| Add deformable branch to forward and backward branches (ablation) | DefMamba tiny model · ImageNet-1K | Top-1 accuracy: 76.9 -> 78.6 | Helped |
| Use only the deformable branch instead of forward and backward branches (ablation) | DefMamba tiny model · ImageNet-1K | Top-1 accuracy: 76.9 -> 76.5 | Hurt |
| Add continuous scanning branch to forward and backward branches (ablation) | DefMamba tiny model · ImageNet-1K | Top-1 accuracy: 76.9 -> 77.3 | Helped |
| Add local scanning branch to forward and backward branches (ablation) | DefMamba tiny model · ImageNet-1K | Top-1 accuracy: 76.9 -> 77.1 | Helped |
| Add deformable points (ablation) | Deformable scanning · ImageNet-1K | Top-1 accuracy: 77.0 -> 77.4 | Helped |
| Add deformable tokens (ablation) | Deformable scanning · ImageNet-1K | Top-1 accuracy: 77.0 -> 77.2 | Helped |
Surprising findings
- Using only the deformable branch without the standard forward/backward branches degrades ImageNet accuracy (ablation Table 4 (1) vs (2)), attributed to increased spatial token jumps.
Limitations
- Deformable scanning becomes less effective for incomplete objects, where offsets shrink toward the default scan order.
- Regularly arranged objects give minimal token-to-token variation, so the model learns slowly in these cases.
- Sorting-based token reordering truncates gradients, so the offset gradient must be approximated heuristically.
Experimental setup
Datasets: ImageNet-1K MSCOCO 2017 ADE20K
Benchmarks: ImageNet-1K MSCOCO 2017 ADE20K
Metrics: Top-1 accuracy box mAP mask mAP mIoU
Baselines: VMamba-T GrootV-T QuadMamba-S LocalVMamba-T MSVMamba-T PlainMamba-L1 ViM-T ConvNeXt-T Swin-T DeiT-S
Builds on: Mamba VMamba deformable attention (DAT) Swin Transformer relative position bias UperNet
Closest papers by meaning
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.87)
- 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification (CVPR 2025 · similarity 0.87)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.87)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.86)
- Boosting Vision State Space Model with Fractal Scanning (AAAI 2025 · similarity 0.85)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.