Research map / 3D vision / 3D generation
Viewpoint-Aware Visual Grounding in 3D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VPP-Net, which explicitly predicts the speaker's viewpoint from a referring expression and 3D scene (pretrained on a templated synthetic viewpoint-annotated dataset) and adds a uniform object representation loss for 3D visual grounding.
Problem
3D visual grounding of referring expressions with viewpoint-dependent spatial language (e.g., "to the right of") is ambiguous because 3D datasets lack speaker viewpoint annotations.
Approach
Generate synthetic ScanNet-based referring expressions with viewpoint (location and heading) labels, train an encoder that predicts the observer's viewpoint and rotates/translates the scene before grounding, plus a uniform object representation loss for viewpoint-invariant object features.
Key results
- VPP-Net reaches 55.65% Acc@0.25 overall on ScanRefer, improving over EDA by 1.06 points.
- VPP-Net achieves 68.7% on SR3D and 56.9% on NR3D (Acc@0.25), +2.00 over the prior best on NR3D.
- Ablation: removing the synthetic pretraining and viewpoint components drops ScanRefer Acc@0.25 to 49.23 to 52.1 range; full model 55.65.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add viewpoint prediction and viewpoint data augmentation (ablation) | VPP-Net · ScanRefer | Acc@0.25: 51.7 -> 55.65 | Helped |
| add viewpoint prediction and viewpoint data augmentation (ablation) | VPP-Net · ScanRefer | Acc@0.50: 38.9 -> 43.29 | Helped |
| add uniform object representation loss (ablation) | VPP-Net · ScanRefer | Acc@0.25: 53.78 -> 55.65 | Helped |
| add uniform object representation loss (ablation) | VPP-Net · ScanRefer | Acc@0.50: 38.98 -> 43.29 | Helped |
| retain curriculum filtering during pretraining (ablation) | VPP-Net · ScanRefer | Acc@0.25: 51.52 -> 55.65 | Helped |
| retain curriculum filtering during pretraining (ablation) | VPP-Net · ScanRefer | Acc@0.50: 38.45 -> 43.29 | Helped |
| add viewpoint data augmentation (ablation) | VPP-Net · ScanRefer | Acc@0.25: 53.78 -> 55.65 | Helped |
| add viewpoint data augmentation (ablation) | VPP-Net · ScanRefer | Acc@0.50: 40.41 -> 43.29 | Helped |
| use synthetic viewpoint pretraining before fine-tuning (ablation) | VPP-Net · ScanRefer | Acc@0.25: 49.23 -> 55.65 | Helped |
| use synthetic viewpoint pretraining before fine-tuning (ablation) | VPP-Net · ScanRefer | Acc@0.50: 33.67 -> 43.29 | Helped |
| use VPP-Net instead of EDA | 3D visual grounding · ScanRefer Overall | Acc@0.25: improving Accuracy@0.25IoU by 1.06% | Helped |
| use VPP-Net instead of EDA | 3D visual grounding · SR3D | Acc@0.25IoU: 0.60% | Helped |
Surprising findings
- Jointly training with viewpoint prediction without first pretraining on synthetic viewpoint data yields worse results than the baseline with no viewpoint modeling (49.23 vs 52.1 Acc@0.25 on ScanRefer).
Limitations
- Synthetic expressions are template-based, limiting diversity and realism.
- Viewpoint generation ignores object visibility from the observer location.
- Gains on SR3D are small due to domain gap between synthetic data and SR3D's proximity-heavy expressions.
Experimental setup
Datasets: ScanNet ScanRefer SR3D NR3D synthetic viewpoint-annotated grounding dataset
Benchmarks: ScanRefer SR3D NR3D
Metrics: Acc@0.25IoU Acc@0.5IoU
Baselines: BUTD-DETR EDA M3DRef-CLIP MVT ViewRefer 3D-SPS UniT3D 3DJCG
Builds on: BUTD-DETR PointNet++ RoBERTa GroupFree
Closest papers by meaning
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.90)
- CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (ICLR 2024 · similarity 0.89)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.89)
- Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency (CVPR 2024 · similarity 0.89)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.89)
- SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.