Research map / Point clouds / Segmentation and detection
ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an open-vocabulary 3D object detector using only 2D images by lifting images to pseudo point clouds with monocular depth and rendering point clouds back to pseudo images via ControlNet, then fine-tunes with optional real 3D data.
Problem
Open-vocabulary 3D object detection is limited by scarce annotated 3D data, and a modality gap between 2D training images and 3D point cloud test inputs prevents direct transfer of 2D open-vocabulary knowledge.
Approach
Unify training images and test point clouds in a pseudo image-point cloud representation: lift 2D images to pseudo point clouds with monocular depth (ZoeDepth) and normal-based rotation correction, generate pseudo 3D boxes from 2D annotations with GPT-4 size priors, and render point clouds back to textured images via a fine-tuned ControlNet for multimodal 3D detection using point cloud input only.
Key results
- Pretraining only on pseudo 3D data reaches 12.61 mAP@0.25 on SUNRGBD and 12.64 on ScanNet, beating the best baseline OV-3DET (5.47 / 5.69).
- With adaptation on real point clouds, mAP@0.25 reaches 22.53 on SUNRGBD and 21.45 on ScanNet, versus OV-3DET's 20.46 and 18.02.
- Ablation: replacing depth maps with ControlNet-rendered pseudo images raises SUNRGBD mAP from 4.38 to 12.61.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with 2D-image-generated pseudo 3D data | ImOV3D · SUNRGBD | mAP@0.25: +7.14 | Helped |
| train with 2D-image-generated pseudo 3D data | ImOV3D · ScanNet | mAP@0.25: +6.78 | Helped |
| pretrain on pseudo 3D data, then adapt with 3D data | ImOV3D · SUNRGBD | mAP@0.25: +2.07 | Helped |
| pretrain on pseudo 3D data, then adapt with 3D data | ImOV3D · ScanNet | mAP@0.25: +2.13 | Helped |
| add train-phase prior size filtering (ablation) | ImOV3D · SUNRGBD | mAP@0.25: +1.65% | Helped |
| add train-phase prior size filtering (ablation) | ImOV3D · ScanNet | mAP@0.25: +1.27% | Helped |
| add rotation correction (ablation) | ImOV3D · SUNRGBD | mAP@0.25: +1.3% | Helped |
| add rotation correction (ablation) | ImOV3D · ScanNet | mAP@0.25: +1.96% | Helped |
| add prior size filtering and rotation correction (ablation) | ImOV3D · SUNRGBD | mAP@0.25: +2.98% | Helped |
| add prior size filtering and rotation correction (ablation) | ImOV3D · ScanNet | mAP@0.25: +3.31% | Helped |
| add inference-phase semantic size filtering (ablation) | ImOV3D with prior size filtering and rotation correction · SUNRGBD | mAP@0.25: +4.26% | Helped |
| add inference-phase semantic size filtering (ablation) | ImOV3D with prior size filtering and rotation correction · ScanNet | mAP@0.25: +4.31% | Helped |
Surprising findings
- Pretraining with no 3D ground-truth annotations already outperforms prior methods that use real 3D training data on the same benchmarks.
Limitations
- Requires dense point clouds for rendered images to help performance.
- Evaluated only on indoor datasets (SUNRGBD and ScanNet) across 20 categories.
Experimental setup
Datasets: LVIS SUNRGBD ScanNet
Benchmarks: SUNRGBD ScanNet
Metrics: mAP@0.25
Baselines: OV-VoteNet OV-3DETR OV-3DET CoDA CoDAv2
Builds on: ImVoteNet OV-3DET Detic ControlNet ZoeDepth
Closest papers by meaning
- Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data (NeurIPS 2024 · similarity 0.91)
- Towards 3D Objectness Learning in an Open World (NeurIPS 2025 · similarity 0.91)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.90)
- Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection (AAAI 2025 · similarity 0.90)
- FM-OV3D: Foundation Model-Based Cross-Modal Knowledge Blending for Open-Vocabulary 3D Detection (AAAI 2024 · similarity 0.89)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.