Research map / Vision-language and multimodal / CLIP and zero-shot
Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a zero-shot, open-vocabulary 3D visual grounding approach that has an LLM generate visual programs (view-independent, view-dependent, functional modules) executed as Python over 3D detections extended with 2D CLIP/ViLT/BLIP-2 correlation.
Problem
3D visual grounding normally needs extensive object-text annotations and a closed vocabulary, and LLM-based reasoning struggles with view-dependent relations and precise distance computation.
Approach
Use in-context prompted LLMs to generate 3D visual programs whose relation modules handle view-dependent relations by projecting objects into a 2D egocentric view, paired with a language-object correlation module that fuses 3D instance segmentation with 2D open-vocabulary models.
Key results
- On ScanRefer validation, the zero-shot visual program reaches 36.4 Acc@0.25 and 32.7 Acc@0.5 overall, beating TGNN and ScanRefer supervised baselines.
- On Nr3D, the method reaches 39.0 overall accuracy, above InstanceRefer (38.8) and ReferIt3DNet (35.6), but below BUTD-DETR (54.6).
- Visual programming with GPT-4 gets 35.4 Acc@0.5 at $4.24 vs 27.5 for dialog with GPT-4 at $62.6 on a 700-sample subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual programming instead of dialog with LLM (ablation) | Zero-shot 3DVG · ScanRefer validation | Acc@0.5: 25.4 -> 32.1 | Helped |
| Replace GPT-3.5 with GPT-4 (ablation) | Visual programming approach · ScanRefer validation | Acc@0.5: 32.1 -> 35.4 | Helped |
| Add view-dependent relation modules (LEFT, RIGHT, FRONT, BEHIND, BETWEEN) (ablation) | Visual programming approach · ScanRefer validation | accuracy: 26.5 -> 39.0 | Helped |
| Add view-independent relation modules (CLOSEST, FARTHEST, LOWER, HIGHER) (ablation) | Visual programming approach · ScanRefer validation | accuracy: 18.8 -> 39.0 | Helped |
| Use 2D-only information instead of combined 2D and 3D information in LOC (ablation) | Zero-shot 3DVG · ScanRefer validation, Overall | Acc@0.25: 36.4 -> 20.0 | Hurt |
| Use 3D-only information instead of combined 2D and 3D information in LOC (ablation) | Zero-shot 3DVG · Nr3D validation, Overall | top-1 accuracy: 39.0 -> 36.7 | Hurt |
| Use CLIP instead of BLIP-2 as the 2D assistance model (ablation) | LOC module · ScanRefer validation, Overall | Acc@0.25: 36.4 -> 35.7 | Hurt |
| Use ViLT instead of BLIP-2 as the 2D assistance model (ablation) | LOC module · ScanRefer validation, Overall | Acc@0.25: 36.4 -> 35.1 | Hurt |
| Replace PointNeXt with PointNet++ (ablation) | Visual programming approach · ScanRefer validation, Overall | accuracy: 39.0 -> 38.2 | Hurt |
| Replace PointNeXt with PointBERT (ablation) | Visual programming approach · ScanRefer validation, Overall | accuracy: 39.0 -> 38.6 | Hurt |
| Use the zero-shot visual programming approach instead of BUTD-DETR | 3D visual grounding · Nr3D validation, Overall | top-1 accuracy: 54.6 -> 39.0 | Hurt |
| Use the zero-shot visual programming approach instead of BUTD-DETR | 3D visual grounding · ScanRefer validation, Overall | Acc@0.5: 39.8 -> 32.7 | Hurt |
Surprising findings
- The zero-shot visual programming approach outperforms several supervised 3DVG baselines despite using no grounding annotations.
- The 2D-only variant performs worst, indicating that 2D image classification struggles with complex scenes and domain gaps.
Limitations
- Accuracy is bottlenecked by LLM generation of correct visual programs, the main error source.
- Object localization and classification in 3D remains a major error source, relying on closed-set 3D instance segmentation for proposals.
- Trails the strongest supervised method BUTD-DETR substantially on ScanRefer and Nr3D.
Experimental setup
Datasets: ScanRefer Nr3D
Benchmarks: ScanRefer Nr3D
Metrics: Acc@0.25 Acc@0.5 top-1 accuracy
Baselines: ScanRefer TGNN InstanceRefer 3DVG-Transformer BUTD-DETR LERF OpenScene ReferIt3DNet
Builds on: VISPROG ViperGPT Mask3D CLIP
Closest papers by meaning
- SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding (CVPR 2025 · similarity 0.92)
- Chain of Semantics Programming in 3D Gaussian Splatting Representation for 3D Vision Grounding (CVPR 2025 · similarity 0.90)
- SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding (NeurIPS 2025 · similarity 0.89)
- From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs (ICML 2025 · similarity 0.89)
- MonoVLM: Monocular 3D Visual Grounding with Vision Language Models (CVPR 2026 · similarity 0.89)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.