Research map / Detection and segmentation / 3D detection and driving
MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MamBEV, a Mamba/SSM-based BEV 3D detection framework with a Spatial Cross Mamba (XQSSM) module for linear-cost cross-attention between BEV queries and multi-camera image features, evaluated on nuScenes.
Problem
Transformer attention in BEV perception from multi-camera images is computationally and memory expensive, and it is unclear how to fuse BEV queries with image features using state space models efficiently for 3D detection.
Approach
Replace transformer cross-attention with an SSM-based cross attention (XQSSM): image features and BEV queries are interleaved into one sequence, queries are merged at position-aware indices from projected reference points, and query tokens with dt=0 leave the hidden state unchanged so only queries read out states, giving linear complexity in input size.
Key results
- MamBEV-Small with 4 frames reaches NDS 0.525 and mAP 0.423 on nuScenes, versus 0.448 NDS for single-frame BEVFormerV1-Small and 0.517 NDS for 4-frame BEVFormerV1-Base.
- XQSSM cross attention scales to a 200x200 BEV grid with 1600x900 images at 51 GFLOPs and 6.5 GB, versus 1,432.5 GFLOPs and >24 GB for dot-product attention.
- Spatial Cross Mamba matches deformable attention closely (NDS 0.3141 vs 0.3140) in the encoder ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MamBEV-Tiny instead of BEVFormerV2-Tiny | nuScenes val, ResNet50, 3 frames · nuScenes val, ResNet50, 3 frames | NDS: 0.397 -> 0.399 | Helped |
| use MamBEV-Small instead of BEVFormerV1-Base | nuScenes val, ResNet101, 4 frames · nuScenes val, ResNet101, 4 frames | NDS: 0.517 -> 0.525 | Helped |
| increase temporal input from 1 to 5 frames (ablation) | MamBEV · nuScenes val, temporal ablation | NDS: 0.3128 -> 0.3874 | Helped |
| increase temporal input from 5 to 8 frames (ablation) | MamBEV · nuScenes val, temporal ablation | NDS: 0.3874 -> 0.3825 | Hurt |
| replace deformable attention with Spatial Cross Mamba (ablation) | MamBEV encoder · nuScenes val, cross-attention ablation | NDS: 0.3140 -> 0.3141 | No clear effect |
| combine Spatial Cross Mamba with deformable cross-attention (ablation) | MamBEV encoder · nuScenes val, cross-attention ablation | NDS: 0.3140 -> 0.3128 | No clear effect |
| use Project query insertion instead of Append (ablation) | MamBEV spatial cross-attention · nuScenes val, query insertion ablation | NDS: 0.3051 -> 0.3128 | Helped |
| use Project query insertion instead of Prepend (ablation) | MamBEV spatial cross-attention · nuScenes val, query insertion ablation | NDS: 0.3073 -> 0.3128 | Helped |
| use both averaging and RMS normalization instead of neither (ablation) | MamBEV query post-processing · nuScenes val, feature normalization ablation | NDS: 0.3110 -> 0.3128 | Helped |
| replace standard self-attention with Hydra α=4 (ablation) | MamBEV BEV self-attention · nuScenes val, BEV self-attention ablation | NDS: 0.2991 -> 0.3128 | Helped |
| replace deformable self-attention with Hydra α=4 (ablation) | MamBEV BEV self-attention · nuScenes val, BEV self-attention ablation | NDS: 0.2992 -> 0.3128 | Helped |
| increase hidden state size from 32 to 256 (ablation) | MamBEV SSM · nuScenes val, hidden state ablation | NDS: 0.3128 -> 0.3164 | No clear effect |
Surprising findings
- Increasing the SSM hidden state size from 16 to 256 or scaling channels yields minimal performance change, suggesting bigger state is not necessarily better.
- The paper's own runtime measurements show only 4.7 vs 3.7 FPS over a transformer baseline, and the authors note their implementation is not optimized, so the kernel does not reflect theoretical complexity.
Limitations
- The SSD kernel was not adapted to the true linear complexity of XQSSM, so measured runtime does not reflect theoretical efficiency.
- Full attention was needed in the decoder, and the best configuration mixes SSM and deformable layers, so pure SSM cross-attention is not always best.
- Scaling state size or channels gave minimal gains, and the paper's results are mostly on nuScenes with models not beating all baselines on every metric.
Experimental setup
Datasets: nuScenes COCO
Benchmarks: nuScenes
Metrics: NDS mAP mATE mASE mAOE mAVE mAAE FPS GFLOPs memory (GB)
Baselines: BEVFormerV1 BEVFormerV2 FCOS3D Focal-PETR DETR3D PolarDETR-T
Builds on: Mamba Mamba-2 Hydra BEVFormer DETR3D deformable attention
Closest papers by meaning
- Improving Bird's Eye View Semantic Segmentation by Task Decomposition (CVPR 2024 · similarity 0.87)
- A Black-Box Evaluation Framework for Semantic Robustness in Bird’s Eye View Detection (AAAI 2025 · similarity 0.86)
- M-BEV: Masked BEV Perception for Robust Autonomous Driving (AAAI 2024 · similarity 0.86)
- Instance-Aware Multi-Camera 3D Object Detection with Structural Priors Mining and Self-Boosting Learning (AAAI 2024 · similarity 0.86)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.86)
- Toward Real-world BEV Perception: Depth Uncertainty Estimation via Gaussian Splatting (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.