Research map / Detection and segmentation / 3D detection and driving
RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RCTrans, a query-based radar-camera 3D detector that densifies sparse radar BEV features with a U-shaped Radar Dense Encoder and decodes boxes with a Pruning Sequential Decoder, reaching SOTA on nuScenes.
Problem
Radar point clouds are sparse and noisy (elevation ambiguity, mostly empty BEV grids), which makes fusing radar and camera features for 3D object detection difficult.
Approach
Densify radar BEV features with a downsample-then-upsample encoder using self-attention at the coarsest scale and skip connections, then use a query decoder that fuses radar and image tokens in two sequential transformer stages per layer, updating query positions and positional embeddings after each layer, with a training-6-layer/inference-3-layer pruning strategy.
Key results
- On nuScenes test with V2-99 backbone, RCTrans reaches 64.7% NDS and 57.8% mAP, outperforming RCBEVDet (63.9% NDS, 55.0% mAP).
- With Swin-T on nuScenes val at 256x704, RCTrans improves NDS by 3.2% and mAP by 2.4% over RCBEVDet at about 5 ms extra latency.
- Pruning training (6 layers in training, 3 at inference) reduces latency from 69.9 ms to 52.2 ms while NDS rises slightly to 58.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use radar and camera modalities instead of a single modality (ablation) | RCTrans · nuScenes val | NDS: ↑2.4 | Helped |
| Add the RDE U-structure (ablation) | RCTrans · nuScenes val | NDS: ↑2.5 | Helped |
| Add RDE skip connections (ablation) | RCTrans · nuScenes val | mAP: ↑2.8 | Helped |
| Add self-attention to the RDE (ablation) | RCTrans · nuScenes val | mAP: ↑0.8 | Helped |
| Add query position updates (ablation) | RCTrans · nuScenes val | mAP: ↑1.5 | Helped |
| Add sequential multimodal fusion (ablation) | RCTrans · nuScenes val | NDS: ↑1.2 | Helped |
| Train with 6 decoder layers and infer with 3 (ablation) | RCTrans · nuScenes val | NDS: ↑0.2 | Helped |
| Train with 6 decoder layers and infer with 3 rather than use 6 at inference (ablation) | RCTrans · nuScenes val | latency (ms): 69.9; 52.2; reducing inference time by 17.7ms | Helped |
| Reduce training decoder layers from 6 to 3 (ablation) | RCTrans · nuScenes val | NDS: 58.6; 56.8 | Hurt |
| Use RCTrans rather than RCBEVDet | RCTrans · nuScenes test | NDS: 63.9; 64.7 | Helped |
| Use RCTrans rather than CRN | RCTrans · nuScenes test | mAP: 57.5; 57.8 | Helped |
| Use RCTrans rather than CRN | RCTrans · nuScenes test tracking | AMOTA: 56.9; 59.6 | Helped |
Surprising findings
- Using 6 decoder layers at training and only 3 at inference gives slightly higher NDS (58.6 vs 58.4) than using all 6 layers, while cutting latency.
- Training with only 3 decoder layers drops NDS to 56.8, showing the depth during training matters.
Limitations
- Sequential fusion uses roughly twice as many transformer layers as a standard decoder, adding inference cost.
- Radar BEV is 128x128 and radar height information is unreliable, so queries use 2D embeddings for radar.
- Some object queries can collapse to the same region without the pruning strategy.
Experimental setup
Datasets: nuScenes DDAD
Benchmarks: nuScenes val nuScenes test
Metrics: NDS mAP latency mATE mAVE AMOTA FP FN IDS
Baselines: CRN RCBEVDet StreamPETR BEVDepth BEVPoolv2 SOLOFusion CenterFusion CRAFT BEVFormer SparseBEV
Builds on: StreamPETR PETR CMT FUTR3D RCBEVDet
Closest papers by meaning
- RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion (CVPR 2025 · similarity 0.89)
- RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection (CVPR 2024 · similarity 0.88)
- HGSFusion: Radar-Camera Fusion with Hybrid Generation and Synchronization for 3D Object Detection (AAAI 2025 · similarity 0.86)
- RobuRCDet: Enhancing Robustness of Radar-Camera Fusion in Bird's Eye View for 3D Object Detection (ICLR 2025 · similarity 0.86)
- TacoDepth: Towards Efficient Radar-Camera Depth Estimation with One-stage Fusion (CVPR 2025 · similarity 0.86)
- RICCARDO: Radar Hit Prediction and Convolution for Camera-Radar 3D Object Detection (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.