Research map / Detection and segmentation / Open-set and few-shot detection
Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an automatic pipeline (open-vocabulary segmentation + region-aware VLM captions + 2D-3D projection) to create Mosaic3D-5.6M, a 5.6M mask-text dataset over 30K scenes, and trains a language-aligned 3D encoder with a lightweight mask decoder for open-vocabulary 3D semantic and instance segmentation.
Problem
Open-vocabulary 3D scene segmentation is limited by the lack of large-scale 3D datasets with precise region masks and rich textual descriptions.
Approach
Generate 3D mask-text pairs by combining Grounded-SAM2 and SEEM masks on multi-view RGB-D frames with Osprey region-level captions, project them to 3D, and train a SparseUNet encoder contrastively plus a Mask3D-style decoder with a caption loss for single-stage open-vocabulary instance segmentation.
Key results
- Mosaic3D-5.6M model reaches 68.1 f-mIoU on ScanNet20, 18.0 on ScanNet++, 13.1 on Matterport3D, and 15.7 on ScanNet200.
- Trained on ScanNet only, Mosaic3D surpasses OV3D by 1.0 on ScanNet20 and RegionPLC by 3.8 on ScanNet200.
- With Mask3D proposals, Mosaic3D reaches 11.8 mAP on ScanNet200 instance segmentation, outperforming OpenIns3D (8.8) and runs at 1.0 s per scene.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale training from ScanNet only to ScanNet, ARKitScenes, ScanNet++, Matterport3D, and Structured3D (ablation) | Mosaic3D · ScanNet200 validation | f-mIoU: 13.0 -> 15.7 | Helped |
| Use Mosaic3D with Mask3D proposals instead of OpenIns3D with Mask3D proposals | 3D-only ScanNet200 instance segmentation · ScanNet200 validation | mAP: 8.8 -> 11.8 | Helped |
| Use Mosaic3D with Segment3D proposals instead of OpenIns3D with Segment3D proposals | 3D-only ScanNet200 instance segmentation · ScanNet200 validation | mAP: 1.7 -> 2.7 | Helped |
| Add a mask decoder trained with Mosaic3D-5.6M and caption merging (ablation) | Mosaic3D · ScanNet200 validation | mAP: 2.7 -> 3.9 | Helped |
| Use Grounded-SAM2 instead of Grounded-SAM for segmentation (ablation) | RegionPLC architecture with Osprey captions · ScanNet200 | f-mIoU: 9.2 -> 10.6 | Helped |
| Use region-level Osprey captioning instead of Ferret captioning (ablation) | Grounded-SAM segmentation with RegionPLC architecture · ScanNet200 | f-mIoU: 9.0 -> 9.2 | Helped |
| Increase training frames from 25K to 125K (ablation) | Grounded-SAM2 segmentation with Osprey captions · ScanNet200 | f-mIoU: 10.6 -> 11.5 | Helped |
| Add SEEM segmentation to Grounded-SAM2 (ablation) | RegionPLC architecture with Osprey captions and 125K frames · ScanNet200 | f-mIoU: 11.5 -> 11.0 | Hurt |
Limitations
- Relies on multi-view 2D VFM and VLM outputs, so caption quality is bounded by those models.
- Zero-shot results drop when class names are anonymized, indicating dependence on class-name leakage in captions.
Experimental setup
Datasets: ScanNet ARKitScenes Matterport3D ScanNet++ Structured3D
Benchmarks: ScanNet20 ScanNet200 Matterport3D ScanNet++
Metrics: f-mIoU f-mAcc mAP mAP50 mAP25
Baselines: OpenScene RegionPLC OV3D SceneVerse OpenIns3D OpenMask3D SAI3D PLA
Builds on: Segment3D RegionPLC Mask3D Osprey Grounded-SAM
Closest papers by meaning
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.92)
- COS3D: Collaborative Open-Vocabulary 3D Segmentation (NeurIPS 2025 · similarity 0.91)
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.91)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.91)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.90)
- XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.