Research map / Detection and segmentation / 3D detection and driving
Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an open-vocabulary monocular 3D detector (OVM3D-Det) using only RGB images by auto-labeling 3D boxes with Grounded-SAM, Unidepth pseudo-LiDAR, adaptive erosion, and LLM-based size priors with box search.
Problem
Open-vocabulary 3D object detection typically requires LiDAR or 3D sensor data for training and deployment, which is costly and limits data scaling; the paper asks whether monocular 3D detectors can be trained from RGB images alone.
Approach
Auto-label 3D boxes from RGB images using open-vocabulary 2D segmentation and zero-shot monocular depth to form pseudo-LiDAR, then refine them with adaptive mask erosion (size-dependent), PCA orientation estimation, and a box search guided by GPT-4 size priors with a ray tracing plus point ratio loss; train Cube R-CNN with a text-alignment head.
Key results
- OVM3D-Det reaches 8.2 novel AP on KITTI and 11.8 novel AP on nuScenes, versus 1.5 and 2.1 for the Cube R-CNN + Grounding DINO baseline (AP_N, not including OVM3D-Det* variant).
- On SUN RGB-D, OVM3D-Det achieves 10.8 AP versus 7.7 for CODA and 7.1 for OV-3DET trained on pseudo-LiDAR.
- Ablation on KITTI: naive framework gets 7.3 AP vs 18.5 AP for the full framework; ray tracing plus point ratio loss gives 18.5 vs 12.2 with ray tracing alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the naive raw pseudo-LiDAR and tight-box pipeline with the proposed pseudo-LiDAR and box refinement (ablation) | OVM3D-Det · KITTI | AP: 7.3 -> 18.5 | Helped |
| Use adaptive mask erosion instead of fixed-kernel erosion (ablation) | OVM3D-Det · KITTI | AP: 16.4 -> 18.5 | Helped |
| Use PCA instead of histogram-based orientation estimation (ablation) | OVM3D-Det · KITTI | AP: 16.7 -> 18.5 | Helped |
| Add category dimension priors to box refinement (ablation) | OVM3D-Det · KITTI | AP: 14.0 -> 18.5 | Helped |
| Replace LLM dimension priors with dataset-statistics priors (ablation) | OVM3D-Det · KITTI | AP: 18.4 vs 18.5 | No clear effect |
| Add point ratio loss to ray tracing loss for box search (ablation) | OVM3D-Det · KITTI | AP: 12.2 -> 18.5 | Helped |
| Vary lower box-prior threshold τ1 from 0.6 through 0.9 (ablation) | OVM3D-Det · KITTI | AP: 17.6; 18.0; 18.5; 18.4 | Mixed |
| Vary upper box-prior threshold τ2 from 1.1 through 1.4 (ablation) | OVM3D-Det · KITTI | AP: 18.5; 18.5; 18.2; 18.1 | Mixed |
| Apply self-training using the previous model’s predictions as pseudo labels | OVM3D-Det · KITTI | AP: 18.5 -> 19.9 | Helped |
| Use OVM3D-Det instead of Cube R-CNN + Grounding DINO | open-vocabulary monocular 3D detection · KITTI novel classes | AP-N: 1.5 -> 8.2 | Helped |
| Use OVM3D-Det instead of Cube R-CNN + Grounding DINO | open-vocabulary monocular 3D detection · nuScenes novel classes | AP-N: 2.1 -> 11.8 | Helped |
| Use OVM3D-Det instead of Cube R-CNN + Grounding DINO | open-vocabulary monocular 3D detection · SUN RGB-D novel classes | AP-N: 1.5 -> 10.0 | Helped |
Surprising findings
- Replacing GPT-4 size priors with dataset-statistics priors yields similar performance (18.4 vs 18.5 AP), suggesting LLM priors are roughly as good as real statistics.
- Directly feeding pseudo-LiDAR to the point-cloud open-vocabulary detector OV-3DET drops it to 3.4 AP on SUN RGB-D, and training OV-3DET on KITTI pseudo-LiDAR yields only 1.3 AP.
Limitations
- Depth estimates for distant objects are inaccurate, causing poor pseudo-labels at long range (AP-Far 4.5).
- Performance may degrade under adverse lighting or weather conditions.
- Relies on the quality of off-the-shelf depth estimation; failure cases include missed distant objects and reflections mistaken for cars.
Experimental setup
Datasets: KITTI nuScenes SUN RGB-D ARKitScenes
Benchmarks: KITTI nuScenes SUN RGB-D ARKitScenes
Metrics: AP3D mAP3D over IoU 0.05-0.50
Baselines: Cube R-CNN + Grounding DINO OV-3DET CODA Oracle Cube R-CNN
Builds on: Cube R-CNN Grounded-SAM Unidepth pseudo-LiDAR OV-3DET
Closest papers by meaning
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.91)
- Learning Occupancy for Monocular 3D Object Detection (CVPR 2024 · similarity 0.89)
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.89)
- MonoDiff: Monocular 3D Object Detection and Pose Estimation with Diffusion Models (CVPR 2024 · similarity 0.88)
- Towards 3D Objectness Learning in an Open World (NeurIPS 2025 · similarity 0.88)
- An Empirical Study of the Generalization Ability of Lidar 3D Object Detectors to Unseen Domains (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.