Research map / 3D vision / Depth and stereo

Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation

NeurIPS 2026 · 3D vision › Depth and stereo · approach: Depth from light field and defocus · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Radar-Modulated Selection (RMS), injecting radar into Mamba's selective scan via zero-initialised perturbations to step size Δ and readout C, plus a multi-view scan pyramid (MVSP), for radar-camera depth estimation (SemoDepth), reporting SOTA on nuScenes and ZJU-4DRadarCam.

Problem

Radar-camera depth estimation must turn ultra-sparse, all-weather metric radar returns into dense per-pixel depth; existing methods fuse radar and image features outside the backbone's sequence operator, and cross-modal Mamba variants keep selection unimodal.

Approach

Radar enters Mamba's selection mechanism directly: zero-initialised radar terms are added to the step size Δ (with a scalar sigmoid gate) and readout C, while B and A stay image-only, so the model equals pretrained image-only Mamba at initialisation; a tiered pyramid matches scan vs FiLM fusion to radar reach per decoder scale.

Key results

• On nuScenes, SemoDepth reduces MAE@0-80m to 1285.2 mm versus 1833.4 mm for TacoDepth (about 29.9% reduction), with 26.8 ms single-frame latency.
• On ZJU-4DRadarCam, SemoDepth gets the best MAE at every range (e.g., 1029.2 mm at 0-50m vs 1120.1 mm for TacoDepth), with gains of roughly 5-8%.
• Ablation: replacing scan tiers with uniform FiLM raises MAE@80 from 1286 to 1564 mm; adding pre-scan fusion on top of RMS gives no gain (1304 vs 1286 mm).

Surprising findings

• Out-of-scan pre-scan fusion added on top of RMS does not improve accuracy, at odds with the usual assumption that more fusion helps.
• On nighttime nuScenes, attention-based fusion (TacoDepth) remains competitive at short range, with SemoDepth trailing on shorter-range metrics.

Limitations

• Only evaluated on front-camera plus automotive radar; transfer to other modality pairs not validated.
• Margin over baselines is smaller on ZJU-4DRadarCam (5-8% MAE) than nuScenes (~30%), and many ablations are from single seeds.
• Robustness to noisy radar returns is untested.

Experimental setup

Datasets: nuScenes ZJU-4DRadarCam

Benchmarks: nuScenes ZJU-4DRadarCam

Metrics: MAE RMSE iMAE iRMSE

Baselines: Lin et al. RC-PDA R4Dyn DORN Singh et al. CaFNet Li et al. FusionMamba TacoDepth

Builds on: Mamba TacoDepth FusionMamba PCA-GM

Closest papers by meaning

• TacoDepth: Towards Efficient Radar-Camera Depth Estimation with One-stage Fusion (CVPR 2025 · similarity 0.89)
• RobuRCDet: Enhancing Robustness of Radar-Camera Fusion in Bird's Eye View for 3D Object Detection (ICLR 2025 · similarity 0.85)
• HGSFusion: Radar-Camera Fusion with Hybrid Generation and Synchronization for 3D Object Detection (AAAI 2025 · similarity 0.85)
• CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection (NeurIPS 2024 · similarity 0.84)
• RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion (CVPR 2025 · similarity 0.84)
• Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.84)