Research map / Detection and segmentation / 3D detection and driving
RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RaCFormer, a query-based radar-camera 3D object detector that samples features from both image and BEV views, using circular query initialization, radar-aware depth prediction, and a ConvGRU-based temporal radar BEV encoder.
Problem
Radar-camera 3D object detection is limited by inaccurate image-to-BEV depth estimation, which misaligns fused BEV features, and by sparse radar BEV features and height errors from automotive radar.
Approach
Replace BEV-level concatenation/cross-attention fusion with a query-based dual-view ray-sampling decoder that samples image-view and BEV features; initialize queries on a linearly increasing circular distribution, refine depth with radar points, and capture radar Doppler motion via a ConvGRU-based implicit dynamic catcher.
Key results
- On nuScenes test set, RaCFormer achieves 64.9% mAP and 70.2% NDS with 6 past and 6 future frames, outperforming HVDetFusion by 4.0% mAP and 2.8% NDS.
- On nuScenes val with ResNet-50 at 256x704, RaCFormer reaches 54.1% mAP and 61.3% NDS, surpassing HyDRa by 4.7% mAP.
- On VoD, RaCFormer achieves 54.44% mAP over the entire annotated area and 78.57% mAP in the region of interest, ranking first.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use query-based radar-camera fusion with dual-view feature sampling | RaCFormer · nuScenes val | mAP: 49.4 -> 54.1 | Helped |
| Use RaCFormer (+6) instead of HVDetFusion (+8) | nuScenes test · nuScenes test | mAP: 60.9 -> 64.9 | Helped |
| Use RaCFormer instead of RCBEVDet | VoD · VoD val, region of interest | mAP: 69.80 -> 78.57 | Helped |
| Add image-view feature sampling to query decoding (ablation) | Query decoder with BEV sampling · nuScenes val | mAP: 40.5 -> 43.6 | Helped |
| Add radar depth and RCS embeddings (ablation) | Radar-aware depth prediction · nuScenes val | mAP: 43.6 -> 44.7 | Helped |
| Add radar depth embedding (ablation) | Radar-aware depth prediction · nuScenes val | mAP: +0.7% | Helped |
| Add radar RCS embedding (ablation) | Radar-aware depth prediction · nuScenes val | mAP: +0.5% | Helped |
| Add the implicit dynamic catcher (ablation) | Camera-radar model · nuScenes val, moving objects >5 m/s | mAP: 43.4 -> 44.9 | Helped |
| Add radar input (ablation) | Camera-only detector · nuScenes val, moving objects (0, 5] m/s | mAP: 12.6 -> 18.8 | Helped |
| Use circular rather than grid query initialization (ablation) | RaCFormer · nuScenes val | mAP: 42.4 -> 43.9 | Helped |
| Set circular query growth factor α=2 (ablation) | Circular query initialization · nuScenes val | mAP: 43.9 -> 43.5 | Hurt |
| Set the number of concentric circles to k=4 (ablation) | Circular query initialization · nuScenes val | mAP: 44.7 -> 44.1 | Hurt |
Surprising findings
- Adding image-view ray sampling on top of BEV query sampling improves mAP by about 3.1%, showing the value of cross-perspective fusion.
Limitations
- Radar height estimation is poor, so depth refinement relies on an assumed default height.
- Standard model uses many queries and multiple frames, making it heavier than real-time variants.
Experimental setup
Datasets: nuScenes View-of-Delft (VoD) nuImages DD3D
Benchmarks: nuScenes View-of-Delft (VoD)
Metrics: mAP NDS mATE mASE mAOE mAVE mAAE AP (3D, KITTI-style)
Baselines: StreamPETR RayFormer HVDetFusion RCBEVDet CRN HyDRa CenterPoint VoxelNeXt RadarPillarNet RCFusion
Builds on: RayFormer BEVFusion PETRv2 SparseBEV lift-splat-shoot
Closest papers by meaning
- RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection (CVPR 2024 · similarity 0.89)
- CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection (NeurIPS 2024 · similarity 0.89)
- RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection (AAAI 2025 · similarity 0.89)
- BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection (CVPR 2024 · similarity 0.88)
- RobuRCDet: Enhancing Robustness of Radar-Camera Fusion in Bird's Eye View for 3D Object Detection (ICLR 2025 · similarity 0.87)
- HGSFusion: Radar-Camera Fusion with Hybrid Generation and Synchronization for 3D Object Detection (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.