Research map / Vision-language and multimodal / Human and embodied modeling
Unifying 2D and 3D Vision-Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniVLG, a unified 2D-3D vision-language grounding model that shares all weights across RGB and RGB-D inputs, uses a language-conditioned mask decoder, and lifts 2D images to pointmaps for joint training, reaching SOTA on sensor-point-cloud 3D referential grounding and 3D QA.
Problem
3D vision-language understanding is limited by scarce large-scale 3D data, and existing 3D grounding models rely on mesh point clouds and ground-truth proposals, which are misaligned with embodied sensor inputs.
Approach
A single architecture initialized from pre-trained 2D ViT weights that processes RGB and RGB-D inputs via shared 3D k-NN attention with pointmap positional embeddings, decodes per-point masks with a language-conditioned Mask2Former-style decoder that updates visual features, and lifts 2D images to pointmaps so all parameters are shared across modalities.
Key results
- On sensor point clouds without GT boxes, UniVLG reaches 73.0 Acc@25 on SR3D, 58.3 on NR3D, and 63.5 on ScanRefer (Det setting), outperforming prior sensor-PC baselines such as 3D-VisTA.
- On ScanQA with sensor point clouds, UniVLG achieves 25.7 EM@1 versus 21.6 for 3D-VisTA; on SQA3D it reaches 50.2.
- Co-training with 2D data improves average 3D grounding accuracy from 61.6 (3D-only) to 64.3 and keeps 2D RefCOCO performance comparable (69.2 vs 69.4 on RefCOCO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| co-train on 2D and 3D data instead of training only on 3D data | UniVLG · 3D referential grounding, SR3D, NR3D, and ScanRefer | Acc@25: 71.6 -> 73.0; 54.7 -> 58.3; 60.7 -> 63.5 | Helped |
| lift 2D images to 3D pointmaps during 2D-3D co-training (ablation) | UniVLG · 3D referential grounding, SR3D, NR3D, and ScanRefer | Acc@25: 73.0 -> 73.0; 53.5 -> 56.3; 61.5 -> 63.5 | Mixed |
| replace the mask decoder with a box decoder (ablation) | UniVLG · 3D grounding, ScanRefer | Acc@25: 60.2 -> 45.7 | Hurt |
| remove visual feature attention during mask decoding (ablation) | UniVLG · 3D grounding, ScanRefer | Acc@25: 60.2 -> 42.8; 54.4 -> 41.5 | Hurt |
| remove pretrained 2D weights (ablation) | UniVLG · 3D grounding, ScanRefer | Acc@25: 60.2 -> 56.9 | Hurt |
| remove mask bounding box loss (ablation) | UniVLG · 3D grounding, ScanRefer | Acc@25: 60.2 -> 56.7 | Hurt |
| use parametric rather than non-parametric queries with a box head (ablation) | UniVLG · 3D grounding, ScanRefer | Acc@25: 34.5 -> 23.9 | Hurt |
| use non-parametric rather than parametric queries with a mask head (ablation) | UniVLG · 3D grounding, ScanRefer | Acc@25: 54.4 -> 43.9 | Hurt |
| add visual feature attention with a box head (ablation) | UniVLG · 3D grounding, ScanRefer | Acc@25: 34.5 -> 33.9 | No clear effect |
| co-train on 2D and 3D data (ablation) | UniVLG · RefCOCO, RefCOCO+, and RefCOCOg val | accuracy: 69.4 -> 69.2; 61.3 -> 61.3; 64.0 -> 64.1 | No clear effect |
| use a DINOv2 rather than Swin visual backbone (ablation) | UniVLG · 3D grounding, SR3D, NR3D, and ScanRefer | Acc@25: 67.1 -> 71.6; 50.6 -> 52.5; 57.3 -> 60.7 | Helped |
| fine-tune rather than freeze the 2D backbone (ablation) | UniVLG · 3D grounding, SR3D, NR3D, and ScanRefer | Acc@25: 66.7 -> 65.6; 52.0 -> 52.7; 54.5 -> 54.4 | Mixed |
Surprising findings
- Box-decoding heads with non-parametric queries perform well on Acc@25 but collapse at Acc@75 (1.1), while mask heads retain 33.2 Acc@75 on ScanRefer.
- Skipping 3D layers for 2D inputs (no lifting) drives 2D-supervised class accuracy in AI2-THOR to nearly zero.
Limitations
- Mask decoding produces distant outlier points and merges multiple instances of the same category, which can inflate boxes derived from masks.
- Only 2D referential grounding data is used beyond RefCOCO family and COCO; scaling 2D data is left for future work.
- Designed for static 3D scenes only.
Experimental setup
Datasets: SR3D NR3D ScanRefer ScanNet200 Matterport3D RefCOCO RefCOCO+ RefCOCOg COCO
Benchmarks: SR3D NR3D ScanRefer ScanQA SQA3D L3DD RefCOCO ScanNet200 Matterport3D
Metrics: Acc@25 Acc@50 Acc@75 EM@1 mAP
Baselines: 3D-VisTA PQ3D BUTD-DETR LLaVA-3D ODIN SAT-2D 3D-LLM NaviLLM LAVT X-Decoder
Builds on: ODIN Mask2Former BUTD-DETR PQ3D DINOv2
Closest papers by meaning
- From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs (ICML 2025 · similarity 0.91)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.90)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.90)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.89)
- UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting (ICLR 2025 · similarity 0.89)
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.