Research map / Vision-language and multimodal / CLIP and zero-shot
SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-shot 3D visual grounding by feeding a 2D VLM (Qwen2-VL-72B) query-aligned rendered images with visual prompts plus text object descriptions, without any 3D training.
Problem
Zero-shot 3D visual grounding: localizing objects in 3D scenes from text without 3D-annotated training data, since 2D VLMs cannot directly consume 3D scenes and text-only LLM approaches miss visual cues.
Approach
Represent a 3D scene as a hybrid of a query-aligned rendered image (viewpoint chosen dynamically from the query's anchor object) with depth-aware visual markers on visible candidate objects, plus a text table of 3D boxes and labels, and let a 2D VLM pick the target ID.
Key results
- On ScanRefer validation, overall Acc@0.25 of 44.1% and Acc@0.5 of 39.4%, the best zero-shot result in the comparison table, versus 36.4/32.7 for ZSVG3D.
- On Nr3D validation, overall accuracy of 46.1%, up from 39.0% for ZSVG3D, and above fully supervised ReferIt3DNet and TGNN.
- Ablation on Nr3D: full configuration reaches 46.1% vs 37.7% with 3D positions only; query-aligned view beats bird's-eye view 46.1% vs 43.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SeeGround instead of ZSVG3D | zero-shot 3D visual grounding · ScanRefer validation, Overall | Acc@0.25: 36.4 -> 44.1 | Helped |
| use SeeGround instead of ZSVG3D | zero-shot 3D visual grounding · ScanRefer validation, Overall | Acc@0.5: 32.7 -> 39.4 | Helped |
| use SeeGround instead of ZSVG3D | zero-shot 3D visual grounding · Nr3D validation, Overall | accuracy: 39.0 -> 46.1 | Helped |
| add scene layout to 3D positions (ablation) | SeeGround without texture, FAM, or PAM · Nr3D validation, Overall | accuracy: 37.7 -> 39.7 | Helped |
| add texture to 3D positions (ablation) | SeeGround without layout, FAM, or PAM · Nr3D validation, Overall | accuracy: 37.7 -> 39.5 | Helped |
| add Fusion Alignment Module (ablation) | SeeGround without Perspective Adaptation Module · Nr3D validation, Overall | accuracy: 39.5 -> 43.3 | Helped |
| add Perspective Adaptation Module (ablation) | SeeGround with layout, texture, and FAM · Nr3D validation, Overall | accuracy: 43.3 -> 46.1 | Helped |
| remove 3D object coordinates (ablation) | SeeGround · Nr3D validation, Overall | accuracy: 46.1 -> 45.0 | Hurt |
| replace query-aligned view with Center2Corner (ablation) | SeeGround · Nr3D validation, Overall | accuracy: 40.2 vs. 46.1 | Hurt |
| replace query-aligned view with Edge2Center (ablation) | SeeGround · Nr3D validation, Overall | accuracy: 41.5 vs. 46.1 | Hurt |
| replace query-aligned view with Corner2Center (ablation) | SeeGround · Nr3D validation, Overall | accuracy: 41.3 vs. 46.1 | Hurt |
| replace query-aligned view with Bird’s Eye View (ablation) | SeeGround · Nr3D validation, Overall | accuracy: 43.3 vs. 46.1 | Hurt |
Surprising findings
- Using the same Qwen2-VL backbone, the authors' method outperforms re-evaluated ZSVG3D, indicating gains are not solely from the VLM choice.
Limitations
- Spatial relationship errors remain high (19% of errors in analysis), and complex viewpoint queries are not handled well.
- Rendering uses dataset point clouds, which limits rendering quality.
- Relies on an open-vocabulary 3D detector; detection errors propagate to grounding.
Experimental setup
Datasets: ScanRefer Nr3D
Benchmarks: ScanRefer Nr3D
Metrics: Acc@0.25 Acc@0.5 accuracy
Baselines: ZSVG3D LLM-Grounder VLM-Grounder OpenScene LERF WS-3DVG MCLN ConcreteNet BUTD-DETR ViL3DRel
Builds on: ZSVG3D LLM-Grounder LAR OpenScene
Closest papers by meaning
- Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding (CVPR 2024 · similarity 0.92)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.90)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.90)
- SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding (NeurIPS 2025 · similarity 0.90)
- DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding (ICLR 2025 · similarity 0.89)
- MonoVLM: Monocular 3D Visual Grounding with Vision Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.