Research map / Vision-language and multimodal / CLIP and zero-shot
Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding
Ruihuang Li, Zhengqiang ZHANG, Chenhang He, Zhiyuan Ma, Vishal Patel, Lei Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| align 3D features with generated text modality in addition to 2D features | DMA (OpenSeg)-3D · ScanNet val | mIoU: 47.9 -> 53.3 | Improved |
| use mutually inclusive BCE loss instead of cross-entropy for text alignment (ablation) | DMA · ScanNet | mIoU: 46.9 -> 53.3 | Improved |
| add MLLM scene descriptions to tagging-model text (ablation) | DMA · ScanNet | mIoU: 46.3 -> 50.5 | Improved |
| add MLLM descriptions to tagging text (ablation) | DMA · nuScenes | mIoU: 43.3 -> 43.8 | No clear effect |
| combine frozen CLIP features with learnable mask features (dual-path) (ablation) | DMA · ScanNet | mIoU: 42.0 -> 44.8 | Improved |
| fine-tune the FC-CLIP mask head instead of freezing it (ablation) | DMA · ScanNet | mIoU: 40.1 -> 42.0 | Improved |
| full mask-head fine-tuning versus LoRA or Adapter (ablation) | DMA · ScanNet | mIoU: 42.0 vs 41.3 (LoRA) vs 40.9 (Adapter) | Improved |
| add GPT denoising of RAM tags (ablation) | DMA · ScanNet | mIoU: +2.6 (basic), +1.3 (enhanced) | Improved |
| use FC-CLIP instead of OpenSeg as 2D encoder | DMA · ScanNet val | mIoU: 53.3 -> 51.8 | Hurt |
| use FC-CLIP instead of OpenSeg as 2D encoder | DMA · nuScenes val | mIoU: 45.1 -> 47.4 | Improved |
| DMA versus fully-supervised model on long-tail ScanNet200 classes | DMA (FC-CLIP)-3D · ScanNet200 tail | mIoU: 3.4 -> 5.8 | Improved |
Closest papers by meaning
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.89)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.89)
- GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields (CVPR 2024 · similarity 0.89)
- Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection (AAAI 2025 · similarity 0.88)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
