Research map / Detection and segmentation / 3D detection and driving
Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoME, a LiDAR-camera 3D detector with three parallel expert decoders (LiDAR, camera, fused) and an Adaptive Query Router that assigns each object query to one expert, trained with synthetic sensor drop for robustness on nuScenes-R and nuScenes-C.
Problem
LiDAR-camera fusion 3D object detectors degrade severely under sensor failures (LiDAR beam reduction/drop, limited FOV, camera drop, occlusion) because features from the two modalities are coupled through inter-modality dependencies.
Approach
Decouple modalities with three parallel DETR-style expert decoders (camera-only, LiDAR-only, fused) and route each object query to exactly one expert via a locality-aware Adaptive Query Router that uses a Local Attention Mask around each query's reference point, keeping decoding cost near a single decoder.
Key results
- On nuScenes-R, MoME reaches 80.5 mAP under LiDAR drop versus 78.4 for CMT, and 55.0 mAP under limited FOV versus 54.9 for CMT, with 71.2 mAP clean.
- Ablation: adding multiple experts alone collapses performance (33.4 mAP under failures), while AQR plus Local Attention Mask raises relative ratio R and improves mAP by 6.4 over the CMT-style baseline; latency rises only 4 ms (274 ms vs 270 ms).
- On nuScenes-C, MoME gets 67.9 mAP under fog versus 66.3 for CMT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a single decoder with multiple expert decoders, without query routing (ablation) | MoME · nuScenes validation clean | mAP: 70.3 -> 33.4 | Hurt |
| Add the Adaptive Query Router to multiple expert decoders (ablation) | MoME · nuScenes validation clean | mAP: 33.4 -> 71.0 | Helped |
| Add the Local Attention Mask to ME+AQR (ablation) | MoME · nuScenes-R | mAP performance ratio: 74.1 -> 80.5 | Helped |
| Use MoME instead of CMT | LiDAR-drop detection · nuScenes-R, LiDAR drop | mAP: +4.2% mAP | Helped |
| Use MoME instead of CMT | limited-FOV detection · nuScenes-R, limited FOV | mAP: +6.7% | Helped |
| Use MoME instead of CMT | camera-drop detection · nuScenes-R, camera view drop | mAP: +1.9% mAP | Helped |
| Use MoME instead of CMT | extreme-weather detection · nuScenes-C fog | mAP: +1.6% | Helped |
| Use MoME instead of CMT | extreme-weather detection · nuScenes-C snow | mAP: +0.9% | Helped |
| Use MoME instead of CMT | extreme-weather detection · nuScenes-C sunlight | mAP: +1.6% | Helped |
| Use MoME instead of the strongest listed baseline | nighttime detection · nuScenes validation, night | mAP: 43.8 vs 43.0 | Hurt |
Surprising findings
- Naively running three parallel expert decoders and picking outputs by confidence substantially hurts performance (clean mAP drops to 71.0 vs 70.3 baseline only with routing; without routing the score is 33.4 mAP under failures), due to uncalibrated confidence scores.
Limitations
- Training only simulates complete sensor drop; partial failures are handled through generalization rather than explicit training.
- Rarely occurring sensor failure patterns are not explicitly simulated during training, as acknowledged in future work.
- Performance under object failure and some scenarios (e.g., occlusion mAP 42.5 vs others) remains limited in absolute terms.
Experimental setup
Datasets: nuScenes nuScenes-R nuScenes-C
Benchmarks: nuScenes-R nuScenes validation nuScenes-C
Metrics: mAP NDS relative performance ratio R
Baselines: CMT UniBEV MetaBEV UniTR SparseFusion BEVFusion DeepInteraction TransFusion CenterPoint DETR3D
Builds on: DETR CMT Mixture of Experts VoxelNet CBGS
Closest papers by meaning
- Layer-Wise Modality Decomposition for Interpretable Multimodal Sensor Fusion (NeurIPS 2025 · similarity 0.87)
- Availability-aware Sensor Fusion via Unified Canonical Space (NeurIPS 2025 · similarity 0.86)
- An Empirical Study of the Generalization Ability of Lidar 3D Object Detectors to Unseen Domains (CVPR 2024 · similarity 0.85)
- AlterMOMA: Fusion Redundancy Pruning for Camera-LiDAR Fusion Models with Alternative Modality Masking (NeurIPS 2024 · similarity 0.85)
- CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation (CVPR 2024 · similarity 0.85)
- L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.