Research map / Detection and segmentation / 3D detection and driving
FM-OV3D: Foundation Model-Based Cross-Modal Knowledge Blending for Open-Vocabulary 3D Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FM-OV3D, which trains a 3DETR-based 3D detector for open-vocabulary detection without human annotations by blending localization boxes from Grounded-SAM with recognition supervision from GPT-3 text prompts, Stable Diffusion images, and CLIP features.
Problem
Open-vocabulary 3D object detection is hard because paired 3D point cloud-text data is scarce and 3D datasets cover few classes; prior methods extract knowledge from single foundation models or depend on annotated 2D/3D data.
Approach
Blend complementary knowledge from several foundation models: Grounded-SAM provides open-vocabulary 3D box supervision for localization, while GPT-3 text prompts and Stable Diffusion images, encoded by CLIP, supply contrastive recognition supervision aligned with point cloud ROI features.
Key results
- Annotation-free FM-OV3D* reaches 16.98 mAP25 on SUN RGB-D and 14.34 on ScanNet, above the prior OV-3DETIC (13.03 and 12.65).
- FM-OV3D (with Detic 2D boxes) reaches 21.47 mAP25 on SUN RGB-D and 21.53 on ScanNet, reported as outperforming OV-3DETIC by 8.44 and 8.88 points.
- Ablation on SUN RGB-D shows GS + GPT-3 + SD gives the best annotation-free mAP25 (16.98) and AR25 (57.22).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Blend GPT-3 text prompts and Stable Diffusion visual prompts instead of using either alone with Detic-generated boxes (ablation) | 3DETR recognition training · SUN RGB-D | mAP25: 16.34 -> 18.19; 18.09 -> 18.19 | Helped |
| Add Stable Diffusion visual prompts to Grounded-SAM boxes and GPT-3 text prompts (ablation) | 3DETR annotation-free training · SUN RGB-D | mAP25: 15.78 -> 16.98 | Helped |
| Add GPT-3 text prompts to Grounded-SAM boxes without Stable Diffusion prompts (ablation) | 3DETR annotation-free training · SUN RGB-D | mAP25: 16.47 -> 15.78 | Hurt |
| Increase GPT-3 prompts per class from 12 to 25 (ablation) | FM-OV3D · SUN RGB-D | mAP25: 11.66 -> 11.70 | Helped |
| Increase GPT-3 prompts per class from 25 to 51 (ablation) | FM-OV3D · SUN RGB-D | mAP25: 11.70 -> 12.50 | Helped |
| Use 5 Stable Diffusion visual-prompt classes rather than 0 (ablation) | FM-OV3D · SUN RGB-D | mAP25: 14.09 -> 15.49 | Helped |
| Increase Stable Diffusion visual-prompt classes from 5 to 7 (ablation) | FM-OV3D · SUN RGB-D | mAP25: 15.49 -> 12.93 | Hurt |
| Use FM-OV3D instead of OV-3DETIC | Open-vocabulary 3D detection · SUN RGB-D | AP25 (mean): 13.03 -> 21.47 | Helped |
| Use FM-OV3D instead of OV-3DETIC | Open-vocabulary 3D detection · ScanNet | AP25 (mean): 12.65 -> 21.53 | Helped |
Surprising findings
- In the annotation-needed setting, GPT-3 text prompts alone or Stable Diffusion images alone already beat the no-prompt baseline, and the combined model is only marginally better (18.19 vs 18.09 mAP25).
- Adding more Stable Diffusion visual prompt classes does not monotonically help; performance declines as the number of classes grows, which the authors attribute to negative-sample noise.
Limitations
- Per-class results are uneven; FM-OV3D underperforms FM-OV3D* on classes such as scanner and ScanNet bookshelf.
- Relies on the 2D grounding pipeline (Grounded-SAM, Detic) and projection matrices provided by datasets.
- Ablations are run only on SUN RGB-D and the number of prompts/visual classes shows non-monotonic trends.
Experimental setup
Datasets: LVIS SUN RGB-D ScanNet
Benchmarks: SUN RGB-D ScanNet
Metrics: mAP25 mAP50 AR25 AR50 per-class AP25
Baselines: GroupFree3D VoteNet H3DNet 3DETR OS-PointCLIP OS-Image2Point Detic-ModelNet Detic-ImageNet OV-3DETIC
Builds on: 3DETR CLIP Grounded-SAM OV-3DETIC Detic
Closest papers by meaning
- Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection (AAAI 2025 · similarity 0.90)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.90)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.89)
- Towards 3D Objectness Learning in an Open World (NeurIPS 2025 · similarity 0.89)
- XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.