Research map / Detection and segmentation / 3D detection and driving
Enhancing 3D Object Detection with 2D Detection-Guided Query Anchors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates 3D query anchors by lifting 2D detections (sampled centers, depth, size, yaw candidates filtered by reprojection IoU) to feed query-based multi-camera 3D detectors, with a shared backbone adapted via visual prompts in two-stage training.
Problem
Query-based multi-camera 3D object detectors miss or mislocalize objects, especially faraway ones, where 2D detectors are more reliable; existing 2D-to-3D lifting methods need separate depth estimation.
Approach
Lift each 2D box to a set of 3D anchors by pairing sampled projected centers with depth, class-specific size and yaw candidates, keep only anchors whose reprojection IoU with the 2D box exceeds a threshold, and use the 2D box class to restrict set-based matching; the 2D and 3D detectors share a backbone via visual prompt tuning.
Key results
- QAF2D improves StreamPETR (V2-99) on nuScenes val by 1.0% NDS and 1.4% mAP over 8DQuery baseline, 1.5% NDS and 1.8% mAP total.
- Largest improvement on nuScenes val is 2.3% NDS and 2.7% mAP, on BEVFormer-small.
- On nuScenes test, StreamPETR-QAF2D (V2-99, 640x1600) reaches 64.2 NDS and 56.6 mAP, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate query anchors from 2D detections instead of random anchors | StreamPETR (V2-99) · nuScenes val | NDS: 57.1 -> 58.6 | Helped |
| generate query anchors from 2D detections instead of random anchors | StreamPETR (V2-99) · nuScenes val | mAP: 48.2 -> 50.0 | Helped |
| generate query anchors from 2D detections | SparseBEV · nuScenes val | mAP: +1.2% mAP | Helped |
| generate query anchors from 2D detections | BEVFormer-small-DAB3D · nuScenes val | NDS: 1.0% NDS | Helped |
| generate query anchors from 2D detections | StreamPETR · nuScenes test | mAP: 0.6% NDS and 1.6% mAP | Helped |
| use Faster-RCNN instead of DAB-DETR as the 2D detector (ablation) | BEVFormer-small-QAF2D · nuScenes val | NDS: 50.2 -> 50.0 | No clear effect |
| add 900 randomly initialized 3D query anchors (ablation) | BEVFormer-small-QAF2D · nuScenes val | NDS: 50.2 -> 50.3 | No clear effect |
| use a separate image feature backbone for the 2D detector instead of sharing (ablation) | BEVFormer-small-QAF2D · nuScenes val | NDS: 49.7 -> 50.3 | Helped |
| add visual prompts when sharing image features (ablation) | BEVFormer-small-QAF2D · nuScenes val | NDS: 49.7 -> 50.2 | Helped |
| set prompt parameter τ to 0.1 (ablation) | BEVFormer-small-QAF2D · nuScenes val | NDS: 49.7 -> 50.0 | Helped |
| set prompt parameter τ to 0.3 (ablation) | BEVFormer-small-QAF2D · nuScenes val | NDS: 49.7 -> 50.1 | Helped |
| set prompt parameter τ to 0.4 (ablation) | BEVFormer-small-QAF2D · nuScenes val | NDS: 49.7 -> 49.8 | No clear effect |
Surprising findings
- Adding random 3D query anchors on top of QAF2D anchors gives only marginal gains (0.1% NDS, 0.4% mAP).
- QAF2D adds about 12 ms of 2D detection overhead, lowering StreamPETR speed from 18.9 to 15.4 FPS.
Limitations
- 3D detection quality depends on the 2D detector; missed 2D objects cannot be recovered by the 3D detector.
- Combining QAF2D anchors with random anchors yields no notable improvement.
- Added 2D detection head increases inference latency.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: NDS mAP mATE mASE mAOE mAVE mAAE
Baselines: StreamPETR SparseBEV BEVFormer DETR3D PolarFormer PETRv2 CAPE MV2D StreamPETR-8DQuery BEVFormer-small-DAB3D
Builds on: DETR StreamPETR SparseBEV BEVFormer DAB-DETR VPT
Closest papers by meaning
- Point2Seq: Detecting 3D Objects As Sequences (CVPR 2022 · similarity 0.85)
- Far3D: Expanding the Horizon for Surround-View 3D Object Detection (AAAI 2024 · similarity 0.84)
- BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection (CVPR 2024 · similarity 0.84)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.83)
- Cross-Modal Distillation for 2D/3D Multi-Object Discovery from 2D Motion (CVPR 2025 · similarity 0.83)
- Multi-View Attentive Contextualization for Multi-View 3D Object Detection (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.