Research map / Detection and segmentation / Open-set and few-shot detection
COS3D: Collaborative Open-Vocabulary 3D Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COS3D, a collaborative prompt-segmentation framework for open-vocabulary 3D Gaussian segmentation that learns an instance field and maps it to a language field, then refines CLIP-based relevance prompts using the instance field.
Problem
Open-vocabulary 3D segmentation with Gaussian splatting suffers from either weak segmentation quality of pure language fields or error accumulation from pre-computed class-agnostic segmentation followed by post-selection.
Approach
A collaborative field pairs a contrastively trained SAM-supervised instance field with a language field obtained by an instance-to-language mapping (shallow MLP or Nadaraya-Watson kernel regression); at inference, the language-field relevance map is used as a prompt that is refined by region growing over the instance field.
Key results
- On LeRF, kernel-regression COS3D reaches 50.76 mean mIoU and 72.08 mean mAcc, versus 38.36 mIoU for OpenGaussian and 43.58 for Dr.Splat.
- On ScanNetv2 (19 classes), kernel-regression COS3D reaches 32.47 mIoU versus 24.73 for OpenGaussian.
- Training takes about 50 minutes per scene on one RTX-4090, versus 60 min for OpenGaussian and 240 min for LangSplat; a 3K-step variant reaches 50.16 mIoU in 8 minutes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use COS3D with kernel regression rather than the strongest reported baseline | LeRF · LeRF mean | mIoU: 45.30 -> 50.76 | Helped |
| use COS3D with kernel regression rather than OpenGaussian | ScanNetv2 · ScanNetv2 (19 classes) | mIoU: 24.73 -> 32.47 | Helped |
| use two-stage instance-to-language mapping rather than joint learning (ablation) | COS3D · LeRF mean | mIoU: 49.15 -> 50.76 | Helped |
| use two-stage instance-to-language mapping rather than independent parallel learning (ablation) | COS3D · LeRF mean | mIoU: 43.84 -> 50.76 | Helped |
| use kernel regression rather than shallow MLP mapping (ablation) | COS3D · LeRF mean | mIoU: 49.75 -> 50.76 | Helped |
| use collaborative prompt inference rather than only the instance branch (ablation) | COS3D · LeRF mean | mIoU: 44.07 -> 50.76 | Helped |
| use collaborative prompt inference rather than only the language branch (ablation) | COS3D · LeRF mean | mIoU: 48.99 -> 50.76 | Helped |
| reduce instance-field training to 3K iterations (ablation) | COS3D · LeRF mean | mIoU: 50.76 -> 50.16 | No clear effect |
| reduce instance-field training to 6K iterations (ablation) | COS3D · LeRF mean | mIoU: 50.76 -> 50.24 | No clear effect |
| replace CLIP with SigLIP as the language model (ablation) | COS3D · LeRF mean | mIoU: 50.76 -> 51.08 | Helped |
| replace SAM with SAM2 for segmentation (ablation) | COS3D · LeRF mean | mIoU: 50.76 -> 51.94 | Helped |
| replace SAM with Semantic SAM for segmentation (ablation) | COS3D · LeRF mean | mIoU: 50.76 -> 49.93 | Hurt |
Surprising findings
- Language-branch-only inference already gives 48.99 mIoU on LeRF, and the instance branch alone gives 44.07, so the combined prompt inference adds a further gain.
- Joint end-to-end learning of the instance and language branches performed worse (49.15 mIoU, 165 min) than the two-stage mapping approach.
Limitations
- Lacks reasoning for relational or multi-object queries because the text-aligned language field is limited.
- Operates in an offline setting; extension to online settings is left open.
Experimental setup
Datasets: LeRF ScanNetv2
Benchmarks: LeRF ScanNetv2
Metrics: mIoU mAcc
Baselines: LangSplat LEGaussians OpenGaussian Dr.Splat InstanceGaussian
Builds on: 3D Gaussian Splatting LangSplat OpenGaussian Segment Anything
Closest papers by meaning
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.91)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.90)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.90)
- Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting (NeurIPS 2025 · similarity 0.89)
- Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation (ICLR 2025 · similarity 0.89)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.