Research map / Detection and segmentation / 3D detection and driving
Availability-aware Sensor Fusion via Unified Canonical Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes availability-aware sensor fusion (ASF) that projects camera, LiDAR and 4D Radar BEV features into a unified canonical space (UCP) and fuses them via cross-attention across sensors along patches (CASAP), trained with a sensor combination loss, achieving SOTA 3D detection on K-Radar.
Problem
Multi-sensor fusion of camera, LiDAR, and 4D Radar for 3D object detection is fragile under sensor degradation or failure, and existing deeply coupled fusion assumes all sensors are available, while sensor-wise cross-attention fusion is computationally expensive and lacks a unified feature represent
Approach
Per-sensor patch-wise projections into a shared canonical space trained against a common reference query, followed by cross-attention only across sensors per patch (O(N_q N_s)) so the network learns to weight available sensors, plus a loss summed over all 7 sensor combinations so one set of weights handles any availability.
Key results
- On K-Radar v1.0 at IoU=0.5, ASF (C+L+R) reaches 87.2% AP_BEV and 73.6% AP_3D, versus 77.5% and 53.5% for prior SOTA L4DR (+9.7 and +20.1 points).
- ASF with L+R runs at 20.5 Hz versus 5.04 Hz for 3D-LRF with the same sensors; C+L+R runs at 13.5 Hz with 1.6 GB VRAM.
- With a damaged camera (C*+L+R), ASF reaches 77.6% AP_3D on v2.0 Sedan vs. 79.3% with all sensors, and the same weights handle all sensor combinations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace L4DR with ASF using all sensors | K-Radar v1.0 · K-Radar v1.0, IoU=0.5, total | AP_3D: 53.5 -> 73.6 | Helped |
| Replace L4DR with ASF using all sensors | K-Radar v1.0 · K-Radar v1.0, IoU=0.5, total | AP_BE V: 77.5 -> 87.2 | Helped |
| Use ASF instead of 3D-LRF with LiDAR and 4D Radar | K-Radar v1.0 · K-Radar v1.0, L+R | FPS: 5.04 -> 20.5 | Helped |
| Remove camera from the full ASF sensor suite | ASF · K-Radar v1.0, IoU=0.5, total | AP_3D: 73.6 -> 72.9 | Hurt |
| Increase patch multiplier from 1 to 4 (ablation) | ASF · K-Radar v2.0, Sedan, IoU=0.3 | AP_3D: 76.1 -> 76.4 | Helped |
| Increase patch multiplier from 1 to 4 (ablation) | ASF · K-Radar v2.0, Bus or Truck, IoU=0.3 | AP_3D: 45.7 -> 47.4 | Helped |
| Reduce patch size from 5 to 2 (ablation) | ASF · K-Radar v2.0, Sedan, IoU=0.3 | AP_3D: 76.4 -> 77.2 | Helped |
| Reduce patch size from 5 to 2 (ablation) | ASF · K-Radar v2.0, Bus or Truck, IoU=0.3 | AP_3D: 47.4 -> 49.7 | Helped |
| Reduce channel dimension from 512 to 256 and increase patch multiplier from 4 to 8 (ablation) | ASF · K-Radar v2.0, Sedan, IoU=0.3 | AP_3D: 77.2 -> 77.6 | Helped |
| Reduce channel dimension from 512 to 256 and increase patch multiplier from 4 to 8 (ablation) | ASF · K-Radar v2.0, Bus or Truck, IoU=0.3 | AP_3D: 49.7 -> 57.9 | Helped |
| Add sensor combination loss (SCL) (ablation) | ASF with P=2, Cu=256, np=8, nh=8 · K-Radar v2.0, Sedan, IoU=0.3 | AP_3D: 77.6 -> 79.3 | Helped |
| Add sensor combination loss (SCL) (ablation) | ASF with P=2, Cu=256, np=8, nh=8 · K-Radar v2.0, Bus or Truck, IoU=0.3 | AP_3D: 57.9 -> 58.2 | Helped |
Surprising findings
- Camera-only AP_3D on K-Radar v2.0 Sedan is only 14.8%, far below radar (47.3%) and LiDAR (73.0%), yet adding camera still helps the fused model slightly.
- ASF with a damaged LiDAR (C+L*+R) often performs similarly to or better than C+R alone.
Limitations
- Camera branch is weak, especially in adverse weather, limiting overall performance.
- Camera encoder trained only on normal or overcast sequences; stereo camera backbone gave no gain over monocular.
- Evaluated on a single dataset (K-Radar).
Experimental setup
Datasets: K-Radar
Benchmarks: K-Radar benchmark v1.0 K-Radar benchmark v2.0
Metrics: AP_BEV AP_3D VRAM FPS
Baselines: RTNH 3D-LRF L4DR DPFT CMT
Builds on: CMT 3D-LRF L4DR DPFT RTNH BEVDepth
Closest papers by meaning
- CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation (CVPR 2024 · similarity 0.87)
- Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion (CVPR 2025 · similarity 0.86)
- CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection (NeurIPS 2024 · similarity 0.85)
- RobuRCDet: Enhancing Robustness of Radar-Camera Fusion in Bird's Eye View for 3D Object Detection (ICLR 2025 · similarity 0.85)
- L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection (AAAI 2025 · similarity 0.85)
- Towards Robust 3D Object Detection with LiDAR and 4D Radar Fusion in Various Weather Conditions (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.