Research map / Detection and segmentation / Open-set and few-shot detection
Segment Any 3D Object with Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOLE, an open-vocabulary 3D instance segmentation framework that predicts semantic-aware masks directly from point clouds using CLIP-fused backbone features, a cross-modality decoder, and three multimodal associations (visual, caption, entity) as supervision, trained without class labels.
Problem
Open-vocabulary 3D instance segmentation with free-form language instructions, where prior methods either overfit to base classes or generate class-agnostic or 2D-projected masks that ignore semantics or geometry.
Approach
Generate semantic-related 3D masks directly from point clouds by fusing projected CLIP (OpenSeg) point features into a Mask3D-style backbone and decoder, and supervise mask embeddings with three levels of multimodal associations (mask-visual, mask-caption via DeCap, mask-entity via noun phrases with soft attention), training with only mask annotations and no class labels.
Key results
- On ScanNetv2 novel classes (7 novel), SOLE reaches 52.3 AP / 72.4 AP50 / 81.7 AP25 versus OpenIns3D's 27.9 AP50 and 42.6 AP25 (2cm voxels).
- On ScanNet200 to Replica cross-domain open-set, SOLE gets 24.7 AP versus OpenMask3D's 13.1 AP.
- On ScanRefer 3D visual grounding with only mask supervision, SOLE gets 25.2 ACC@25 versus 3D-LLM's 21.2, and 22.6 ACC@50 versus ScanRefer's 27.4 fully supervised.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SOLE instead of OpenIns3D | Open-vocabulary 3D instance segmentation · ScanNetv2 novel classes (-/7) | AP50: 27.9 -> 72.4 | Helped |
| Use SOLE instead of OpenMask3D | Open-vocabulary 3D instance segmentation · ScanNet200 | AP: 15.4 -> 20.1 | Helped |
| Use SOLE instead of OpenMask3D | Open-vocabulary 3D instance segmentation · ScanNet200 tail classes | APtail: 14.9 -> 14.1 | Hurt |
| Use SOLE instead of OpenMask3D | Open-vocabulary 3D instance segmentation · ScanNetv2 to ScanNet200 novel classes | AP: 11.9 -> 19.1 | Helped |
| Use SOLE instead of OpenMask3D | Cross-domain 3D instance segmentation · ScanNet200 to Replica | AP: 13.1 -> 24.7 | Helped |
| Use SOLE with mask-only training | 3D visual grounding · ScanRefer | ACC@25: 21.2 -> 25.2 | Helped |
| Use SOLE with mask-only training | 3D visual grounding · ScanRefer | ACC@50: 22.6 vs. 27.4 | Mixed |
| Increase voxel size from 2cm to 4cm (ablation) | SOLE · ScanNetv2 novel classes (-/17) | AP: 44.4 -> 30.8 | Hurt |
| Remove mask-caption and mask-entity supervision (ablation) | SOLE · ScanNetv2 novel classes (-/17) | AP: 44.4 -> 35.0 | Hurt |
| Add both projected 2D CLIP features and 3D backbone features (ablation) | SOLE · ScanNetv2, multimodal fusion, 4cm voxel | AP: 18.7 -> 30.8 | Helped |
| Add Cross Modality Decoder (ablation) | SOLE · ScanNetv2, multimodal fusion, 2cm voxel | AP: 42.8 -> 44.4 (+1.6%) | Helped |
| Use mask-entity association rather than mask-visual association alone (ablation) | SOLE · ScanNetv2, multimodal associations, 4cm voxel | AP: 24.5 -> 32.1 | Helped |
Surprising findings
- Adding mask-visual and mask-caption associations to mask-entity association lowers AP and AP50 while improving AP25, which the authors attribute to semantic learning impairing mask accuracy.
- Removing all text supervision (mask-caption and mask-entity) still yields state-of-the-art results, suggesting the framework does not depend on the captioning model or NLP tools.
Limitations
- Performance on 2cm voxels is far from the fully supervised Mask3D on AP (44.4 vs 55.2 on ScanNetv2 novel setting, and not directly comparable across splits).
- Adding semantic associations trades off mask accuracy at AP50 in the benchmark setting, and the paper notes this tradeoff explicitly.
- Caption and entity supervision depends on the DeCap captioning model and TextBlob/spaCy noun extraction, which may produce errors.
Experimental setup
Datasets: ScanNetv2 ScanNet200 Replica ScanRefer
Benchmarks: ScanNetv2 ScanNet200 Replica ScanRefer
Metrics: AP AP50 AP25 ACC@25 ACC@50
Baselines: PLA RegionPLC Lowis3D OpenIns3D OpenMask3D Mask3D
Builds on: Mask3D OpenSeg DeCap OpenMask3D CLIP
Closest papers by meaning
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.94)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.92)
- XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation (NeurIPS 2024 · similarity 0.91)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.91)
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.90)
- MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.