Research map / Vision-language and multimodal / Human and embodied modeling
Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills multiple 2D foundation models (Lseg, DINOv2, Stable Diffusion) into a 3D sparse-conv backbone with per-model learned uncertainty weights for open-vocabulary 3D segmentation on ScanNetV2 and Matterport3D.
Problem
Open-vocabulary 3D scene understanding usually distills knowledge from a single VLM, which limits the 3D model's use of complementary spatial and semantic knowledge in other foundation models, and noisy multi-view 2D features hurt distillation.
Approach
Use independent projection heads per 2D foundation model, weight each distillation loss by a learned deterministic (Gaussian-likelihood style) observation-noise scalar, and de-mean the heavy-tailed Stable Diffusion features before distillation.
Key results
- On ScanNetV2 val, CUA-O3D (2D3D) reaches 55.3 mIoU vs 52.2 for OpenScene reimplementation, and 54.1 mIoU for 3D-only.
- Ablation: adding Lseg+DINOv2+SD, uncertainty weighting and de-mean raises zero-shot mIoU from 51.4 to 54.1 on ScanNetV2.
- Cross-dataset: ScanNetV2→Matterport3D mIoU 37.4 vs 36.0 for OpenScene; Matterport3D→ScanNetV2 38.6 vs 36.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| agglomerate Lseg, DINOv2, and Stable Diffusion features with uncertainty-aware distillation | MinkUNet18A · ScanNetV2 val | mIoU: 51.6 -> 54.1 | Helped |
| agglomerate Lseg, DINOv2, and Stable Diffusion features with uncertainty-aware distillation | MinkUNet18A · Matterport3D val | mIoU: 40.5 -> 41.3 | Helped |
| train with three foundation-model heads instead of the baseline initialization | CUA-O3D · ScanNetV2 val linear probing | mIoU: 54.4 -> 62.1 | Helped |
| train with three foundation-model heads instead of the baseline initialization | CUA-O3D · Matterport3D val linear probing | mIoU: 36.1 -> 45.8 | Helped |
| add DINOv2 supervision to Lseg supervision (ablation) | CUA-O3D · ScanNetV2 val | mIoU: 51.4 -> 51.7 | Helped |
| add Stable Diffusion supervision to Lseg supervision (ablation) | CUA-O3D · ScanNetV2 val | mIoU: 51.4 -> 51.4 | No clear effect |
| add Stable Diffusion supervision to Lseg and DINOv2 (ablation) | CUA-O3D · ScanNetV2 val | mIoU: 51.7 -> 52.7 | Helped |
| add deterministic uncertainty estimation (ablation) | CUA-O3D with three-model supervision · ScanNetV2 val | mIoU: 52.7 -> 53.5 | Helped |
| use auto-weighting in place of the proposed uncertainty approach (ablation) | CUA-O3D · ScanNetV2 val | mIoU | Hurt |
| add de-mean operation for Stable Diffusion features (ablation) | CUA-O3D · ScanNetV2 val | mIoU: 53.5 -> 54.1 | Helped |
| train on ScanNetV2 and evaluate on Matterport3D | CUA-O3D · ScanNetV2 → Matterport3D | mIoU: 36.0 -> 37.4 (+1.4) | Helped |
| train on Matterport3D and evaluate on ScanNetV2 | CUA-O3D · Matterport3D → ScanNetV2 | mIoU: 36.5 -> 38.6 (+2.1) | Helped |
Surprising findings
- Learning the uncertainty weights automatically (auto-weighting without the regularized form) caused training collapse.
- DINOv2 alone as a linear-probe head performs nearly as well as the Lseg head that was pre-aligned to text (61.7 vs 59.9 mIoU on ScanNetV2 from the 3-head comparison of single heads).
Limitations
- Only three 2D foundation models were integrated, and the uncertainty is applied as a scalar per model rather than per point.
- Training uses randomly initialized MinkUNet18A without RGB input and samples only 20k points due to GPU memory limits.
- Gains over OpenScene are modest (e.g., +0.8 to +2.5 mIoU) and mainly reported on zero-shot settings.
Experimental setup
Datasets: ScanNetV2 Matterport3D
Benchmarks: ScanNetV2 Matterport3D
Metrics: mIoU mAcc
Baselines: OpenScene RegionPLC CLIP2Scene DMA CNS PLA CLIP-FO3D MSeg
Builds on: OpenScene MinkowskiNet AM-RADIO Probe3D
Closest papers by meaning
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.91)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.90)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.90)
- FM-OV3D: Foundation Model-Based Cross-Modal Knowledge Blending for Open-Vocabulary 3D Detection (AAAI 2024 · similarity 0.90)
- GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields (CVPR 2024 · similarity 0.90)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.