Research map / Detection and segmentation / Open-set and few-shot detection
Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Open-YOLO 3D replaces SAM and CLIP in open-vocabulary 3D instance segmentation with YOLO-World 2D detections projected onto 3D proposals (MVPDist, LG label maps, GPU visibility), achieving SOTA mAP with about 16x speedup over Open3DIS.
Problem
Open-vocabulary 3D instance segmentation methods rely on SAM and CLIP for multi-view 3D feature aggregation, making inference take minutes per scene and limiting real-time use.
Approach
Label 3D class-agnostic Mask3D proposals by projecting them onto low-granularity label maps built from open-vocabulary 2D detector boxes (YOLO-World), aggregated over top-k visible views into a multi-view prompt distribution, plus a mask score that multiplies a 2D-box IoU term by the class probability.
Key results
- On ScanNet200 val, Open-YOLO 3D reaches 24.7 mAP at about 21.8 s per scene, versus 23.7 mAP for Open3DIS at 360 s with 2D-mask proposals.
- On Replica with 3D-network proposals, it reaches 23.7 mAP versus 14.9 for Open3DIS, at 16.6 s per scene versus 35.08 s.
- With ground-truth masks on ScanNet200, MVPDist reaches 39.6 mAP versus 30.9 for Open3DIS using point-wise CLIP features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Open-YOLO 3D instead of Open3DIS with 3D proposals only | ScanNet200 val · ScanNet200 val | mAP: 18.6 -> 24.7 | Helped |
| use Open-YOLO 3D instead of Open3DIS with 3D proposals only | ScanNet200 val · ScanNet200 val | time/scene (s): 57.68 -> 21.8 | Helped |
| use Open-YOLO 3D instead of Open3DIS with 3D proposals only | Replica · Replica | mAP: 14.9 -> 23.7 | Helped |
| use Open-YOLO 3D instead of Open3DIS with 3D proposals only | Replica · Replica | time/scene (s): 35.08 -> 16.6 | Helped |
| replace point-wise CLIP matching with MVPDist | 3D mask labeling with oracle masks · ScanNet200 with oracle masks | mAP: 30.9 -> 39.6 | Helped |
| replace SAM with YOLOv8 for crop generation, retaining CLIP features (ablation) | Replica labeling with ground-truth masks · Replica with ground-truth masks | mAP: 33.0 -> 21.1 | Hurt |
| replace SAM with RT-DETR for crop generation, retaining CLIP features (ablation) | Replica labeling with ground-truth masks · Replica with ground-truth masks | mAP: 33.0 -> 28.4 | Hurt |
| replace SAM with YOLO-World for crop generation, retaining CLIP features (ablation) | Replica labeling with ground-truth masks · Replica with ground-truth masks | mAP: 33.0 -> 32.5 | No clear effect |
| replace CLIP features with MVPDist and LG label maps (ablation) | YOLO-World labeling · Replica with ground-truth masks | mAP: 32.5 -> 46.2 | Helped |
| add accelerated visibility computation (VAcc) (ablation) | YOLO-World + MVPDist · Replica with ground-truth masks | time/scene (s): 376.42 -> 17.86 | Helped |
| use HG rather than LG label maps (ablation) | Open-YOLO 3D, top-k 40 · ScanNet200 val | mAP: 24.7 -> 24.5 | Hurt |
| use top-k 40 views rather than top-k 1 (ablation) | Open-YOLO 3D with LG label maps · ScanNet200 val | mAP: 16.3 -> 24.7 | Helped |
Surprising findings
- Replacing SAM with YOLO-World for label maps keeps accuracy: low-granularity label maps match or beat high-granularity SAM-based maps (24.7 vs 24.5 mAP at top-40 in the table) at about 5x lower time.
- A 2D object detector with multi-view label distributions outperforms CLIP feature aggregation for matching text prompts to 3D masks.
Limitations
- Depends on a 3D proposal network (Mask3D) that misses small objects, so recall of small geometries is limited.
- Performance depends on the open-vocabulary 2D detector making consistent class predictions across views.
- Uses 3D proposals only; 2D-mask-based proposal fusion from fast segmenters like FastSAM is left as future work.
Experimental setup
Datasets: ScanNet200 Replica
Benchmarks: ScanNet200 Replica
Metrics: mAP mAP50 mAP25 time per scene
Baselines: OpenMask3D Open3DIS OVIR-3D SAM3D OpenScene Mask3D
Builds on: OpenMask3D Mask3D YOLO-World Open3DIS
Closest papers by meaning
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.91)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.89)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.89)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.89)
- COS3D: Collaborative Open-Vocabulary 3D Segmentation (NeurIPS 2025 · similarity 0.89)
- YOLO-World: Real-Time Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.