Research map / Vision-language and multimodal / Human and embodied modeling

Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding

AAAI 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Region-level vision-language alignment · 28 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes 3DVLP, a 3D vision-language pre-training framework using visual grounding as proxy task plus object-level IoU-guided detection, cross-contrastive and self-contrastive losses, evaluated on ScanRefer, Scan2Cap and ScanQA.

Problem

Task-specific 3D scene understanding models (visual grounding, dense captioning, question answering on point clouds) do not yield universal 3D vision-language representations, and existing 3D VL pre-training objectives do not transfer well to these tasks.

Approach

Pre-trains a VoteNet-based point cloud encoder with visual grounding as the proxy task, adding an IoU-filtered label-smoothed DIoU detection loss and object-level contrastive losses (cross-modal OCC and intra-modal OSC) where all proposals above an IoU threshold count as positives.

Key results

• 3DVLP reaches 51.70% Acc@0.25 and 40.51% Acc@0.5 overall on ScanRefer, versus 49.68/38.08 for 3DLP (3D-only setting: 3DLP 49.68/38.08 vs. BUTD-DETR 49.69/38.01).
• On ScanQA, 3DVLP achieves EM@1 24.03 and EM@10 57.91, versus 21.05 and 51.23 for the ScanQA baseline.
• On Scan2Cap at IoU 0.5, 3DVLP obtains M@0.5 34.34 and R@0.5 54.28, a clear gain on METEOR over 3DLP's 24.83.

Measured effects reported in the paper

Surprising findings

• Adding OSC or OCC alone gives little gain in Acc@0.25, while the OID loss mainly drives the improvement in Acc@0.5.

Limitations

• Pre-training uses only visual grounding as proxy task and a frozen BERT language encoder, limiting full multimodal interaction.
• IoU filter threshold must be tuned (0.25 chosen); performance drops at high thresholds and is sensitive to threshold choice.

Experimental setup

Datasets: ScanRefer Scan2Cap ScanQA ScanNet

Benchmarks: ScanRefer Scan2Cap ScanQA

Metrics: Acc@0.25 Acc@0.5 C@0.5 B-4@0.5 M@0.5 R@0.5 EM@1 EM@10

Baselines: 3DVG-Transformer 3D-SPS BUTD-DETR 3DJCG 3DLP D3Net MORE SpaCap3D FE-3DGQA ScanQA

Builds on: VoteNet ScanRefer 3DLP DIoU loss label smoothing

Closest papers by meaning

• RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding (CVPR 2024 · similarity 0.91)
• Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception (CVPR 2024 · similarity 0.89)
• Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.89)
• Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.89)
• VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completion (AAAI 2025 · similarity 0.89)
• Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.88)