Research map / 3D vision / 3D generation
A Unified Framework for 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniSeg3D, a single transformer-based model that unifies six 3D segmentation tasks (panoptic, semantic, instance, interactive, referring, open-vocabulary) and links them via knowledge distillation and ranking-based contrastive learning.
Problem
Existing 3D scene segmentation methods are specialized for individual tasks (generic, open-vocabulary, interactive, referring), missing cross-task knowledge and requiring separate models for each.
Approach
A unified query-based mask decoder processes generic, point-prompt (superpoint features), and text-prompt queries with task embeddings, and explicit inter-task associations (distillation from interactive to generic/referring segmentation plus vision-text ranking contrastive loss) are used only during training.
Key results
- UniSeg3D reaches 71.3 PQ on ScanNet20 panoptic segmentation, 0.1 above OneFormer3D (71.2), and 76.9 mIoU semantic with the two-stage fine-tuning trick.
- On ScanRefer referring segmentation, UniSeg3D achieves 29.5 mIoU (29.6 with fine-tuning) using 72.4% of the training data, close to X-RefSeg3D's 29.9.
- Open-vocabulary AP on ScanNet200 is 19.6 (19.7 with fine-tuning), above Open3DIS (19.0); the ablation shows the full model achieves Overall 51.7 vs 51.3 baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add knowledge distillation from interactive segmentation (ablation) | UniSeg3D · ScanNet20 instance segmentation | mAP: 58.0 -> 58.6 | Helped |
| add knowledge distillation from interactive segmentation (ablation) | UniSeg3D · ScanNet20 interactive segmentation | AP: 54.5 -> 55.3 | Helped |
| add rank-contrastive learning (ablation) | UniSeg3D · ScanRefer | mIoU: 29.1 -> 29.6 | Helped |
| add rank-contrastive learning (ablation) | UniSeg3D · ScanNet20 interactive segmentation | AP: 54.5 -> 54.1 | Hurt |
| combine knowledge distillation with rank-contrastive learning (ablation) | UniSeg3D · Overall | Overall: 51.3 -> 51.7 | Helped |
| add the two-stage fine-tuning trick (ablation) | UniSeg3D · Overall | Overall: 51.7 -> 51.9 | Helped |
| use the ranking rule in contrastive learning (ablation) | UniSeg3D contrastive learning · ScanNet20 instance segmentation | mAP: 59.0 -> 59.1 | Helped |
| set lambda to 0.2 instead of 0.1 (ablation) | UniSeg3D · ScanNet20 interactive segmentation | AP: 54.1 -> 52.3 | Hurt |
| set lambda to 0.3 instead of 0.1 (ablation) | UniSeg3D · Overall | Overall: 51.7 -> 50.9 | Hurt |
| place the vision prompt at the instance center rather than use a random point (ablation) | UniSeg3D interactive segmentation · ScanNet20 | AP: 51.3 -> 56.6 | Helped |
| add interactive, referring, and open-vocabulary tasks to the generic-task framework (ablation) | UniSeg3D unification baseline · ScanNet20 panoptic segmentation | PQ: 71.0 -> 70.3 | Hurt |
| unify six tasks in one model rather than use task-specialized approaches | UniSeg3D · ScanNet20 panoptic segmentation | PQ: +12.1 PQ | Helped |
Surprising findings
- Naively adding interactive, referring, and open-vocabulary tasks to the unified model degrades performance on panoptic, instance, and interactive segmentation, showing task-balancing difficulty.
- Interactive segmentation AP drops by up to 20.2 when the vision prompt lies at object edges rather than centers.
Limitations
- Only evaluated on indoor scenes; outdoor scenes unexplored.
- Interactive segmentation performance drops when vision prompts are far from instance centers.
- Referring training uses a filtered ScanRefer subset (72.4%) to avoid open-vocabulary label leakage, so comparison with full-data baselines is not like-for-like.
Experimental setup
Datasets: ScanNet20 ScanNet200 ScanRefer Replica
Benchmarks: ScanNet20 ScanRefer ScanNet200 Replica
Metrics: PQ mIoU mAP AP AP50 AP25 acc@0.25
Baselines: OneFormer3D SPFormer Mask3D MAFT AGILE3D X-RefSeg3D Open3DIS OpenMask3D SOLE OpenScene PanopticNDT OctFormer
Builds on: OneFormer3D OneFormer CLIP Open3DIS SAM3D
Closest papers by meaning
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.89)
- Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.89)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.89)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.