Research map / Detection and segmentation / 3D detection and driving
Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cocoon, an object- and feature-level uncertainty-aware multi-modal (camera-LiDAR) 3D detection fusion framework that uses conformal prediction with a feature aligner and learnable feature impressions (geometric-median surrogate ground truths) to weight modalities adaptively.
Problem
Adaptive camera-LiDAR fusion for 3D object detection is hard under long-tail and corrupted conditions because existing MoE fusion ignores object-level uncertainty and late fusion relies on separate detectors; uncertainty is hard to compare across heterogeneous modality features at intermediate layer
Approach
Align heterogeneous 2D and 3D query features into a common space with an MLP feature aligner, learn feature impression nodes as geometric-median surrogate ground truths, and derive a conformal-prediction nonconformity score to compute per-query, per-modality uncertainty weights for linear fusion.
Key results
- On nuScenes under the blackout camera corruption, Cocoon with FUTR3D reaches 51.87 mAP vs. 45.13 for the static base model (about +15% relative in mAP as claimed in the intro).
- Cocoon achieves 66.80 mAP on the clean nuScenes validation set vs. 66.16 for the static FUTR3D base and 62.33 for AdapNet.
- In the coverage check on 10 datasets at alpha=0.1, Cocoon's nonconformity function gives mean coverage closest to 90% on most datasets (e.g., Meps20 90.02, Star 90.02).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace static 1:1 fusion with Cocoon uncertainty-aware adaptive fusion | FUTR3D · nuScenes val, no corruption | mAP: 66.16 -> 66.80 | Helped |
| replace static 1:1 fusion with Cocoon uncertainty-aware adaptive fusion | FUTR3D · nuScenes natural corruptions (rainy day, clear night, rainy night) | mAP: 68.34 -> 68.89; 44.51 -> 45.68; 27.26 -> 27.98 | Helped |
| replace static 1:1 fusion with Cocoon uncertainty-aware adaptive fusion | FUTR3D · nuScenes artificial corruptions (point sampling, random noise, blackout, sensor misalignment) | mAP: 65.17 -> 65.89; 60.39 -> 61.83; 45.13 -> 51.87; 53.16 -> 55.61 | Helped |
| replace static fusion with Cocoon uncertainty-aware adaptive fusion | TransFusion · nuScenes val, no corruption (120 samples) | mAP, total: +0.7 | Helped |
| replace static fusion with Cocoon uncertainty-aware adaptive fusion | TransFusion · nuScenes val, misalignment (120 samples) | mAP, total: 68.6 -> 68.5 | Hurt |
| replace static fusion with Cocoon uncertainty-aware adaptive fusion | TransFusion · nuScenes val, blackout and sampling (120 samples) | mAP, total: +0.1; +1.0 | Helped |
| replace feature concatenation with element-wise sum (ablation) | FUTR3D · proper-training-set evaluation | mAP: 67.39% -> 67.39% | No clear effect |
| compute nonconformity only on the last three decoder layers (ablation) | Cocoon on FUTR3D · nuScenes no corruption and blackout (C) | mAP: 66.28%; 51.3% | Mixed |
| integrate Cocoon rather than use static fusion | FUTR3D · inference | latency (seconds): 1.1 ± 0.1 -> 1.6 ± 0.1 | Hurt |
| integrate Cocoon rather than use static fusion | TransFusion · inference | latency (seconds): 0.7 ± 0.1 -> 0.7 ± 0.1 | No clear effect |
Surprising findings
- Latency with FUTR3D rises by about 0.5 s (1.1 to 1.6 s) because nonconformity scores are computed in all six decoder layers, a notable runtime overhead.
- Many FUTR3D queries (up to 900 predefined, only ~200 needed) are meaningless, and high alpha values degrade output, requiring a clipping strategy.
Limitations
- Runtime overhead of about 0.5 s on FUTR3D due to nonconformity computation across six decoder layers.
- Meaningless queries in FUTR3D (900 predefined vs. ~200 needed) degrade fusion when alpha is large, requiring clipping.
- Base models must be retrained on a proper training split (calibration data unavailable for full training), and TransFusion evaluation uses only 120 validation samples.
Experimental setup
Datasets: nuScenes UCI Machine Learning Repository datasets (bio, bike, community, facebook1, facebook2) Blog Feedback STAR MEPS19 MEPS20 MEPS21
Benchmarks: nuScenes validation set nuScenes under natural corruptions (rainy day, clear night, rainy night) nuScenes under artificial corruptions (blackout, random noise, point sampling, sensor misalignment) 10 uni-dimensional real-world datasets for coverage
Metrics: mAP empirical coverage absolute difference from 90% target inference latency
Baselines: Base FUTR3D AdapNet Self-attention adaptive fusion Late fusion with enhanced NMS Cal-DETR (T) Cal-DETR (Q)
Builds on: FUTR3D TransFusion Feature CP Cal-DETR Weiszfeld's algorithm
Closest papers by meaning
- RobuRCDet: Enhancing Robustness of Radar-Camera Fusion in Bird's Eye View for 3D Object Detection (ICLR 2025 · similarity 0.85)
- Hyperdimensional Uncertainty Quantification for Multimodal Uncertainty Fusion in Autonomous Vehicles Perception (CVPR 2025 · similarity 0.85)
- UGG-ReID: Uncertainty-Guided Graph Model for Multi-Modal Object Re-Identification (NeurIPS 2025 · similarity 0.85)
- SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection (AAAI 2024 · similarity 0.85)
- Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion (CVPR 2025 · similarity 0.85)
- FusionFormer: A Concise Unified Feature Fusion Transformer for 3D Pose Estimation (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.