Research map / 3D vision / Gaussian splatting
Chain of Semantics Programming in 3D Gaussian Splatting Representation for 3D Vision Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a zero-shot neuro-symbolic 3D visual grounding framework that uses GPT-4o to parse utterances into a relationship graph and chain of semantics, explores a 3D Gaussian Splatting reconstruction to render viewpoint images, generates Python code, and self-checks execution errors.
Problem
Zero-shot 3D vision grounding struggles with fine-grained semantics from sparse point clouds and with complex, conditional spatial relationships in utterances, especially view-dependent ones.
Approach
Decompose the utterance into a relationship graph and a chain of semantics (step-wise conditional grounding steps), guide LLM code generation with it, render viewpoint-specific 2D images from a 3DGS reconstruction for extra semantics, and use a grounded-aware self-check that feeds execution errors back to the programmer.
Key results
- Achieves 60.8% overall accuracy on Nr3D, surpassing prior zero-shot methods (VLM-Grounder 48.0%).
- Achieves 91.4% overall accuracy on Sr3D, above supervised baselines such as CoT3DRef (73.6%) and 3D-VisTA (76.4%).
- 3DGS rendering improves view-dependent accuracy on Sr3D from 37.5% to 52.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add grounded-aware self-check (ablation) | Chain of Semantics Programming · Nr3D | accuracy: 56.3% -> 60.8% | Helped |
| add grounded-aware self-check (ablation) | Chain of Semantics Programming · Sr3D | accuracy: 89.6% -> 91.4% | Helped |
| add chain of semantics to dialogue grounding (ablation) | GPT-4o dialogue grounding · Nr3D | accuracy: 43.0% -> 45.2% | Helped |
| add chain of semantics to dialogue grounding (ablation) | GPT-4o dialogue grounding · Sr3D | accuracy: 64.3% -> 60.3% | Hurt |
| use programming rather than dialogue grounding (ablation) | GPT-4o grounding · Nr3D | accuracy: 43.0% -> 53.2% | Helped |
| use programming rather than dialogue grounding (ablation) | GPT-4o grounding · Sr3D | accuracy: 64.3% -> 84.8% | Helped |
| add chain of semantics to programming (ablation) | GPT-4o programming grounding · Nr3D | accuracy: 53.2% -> 55.1% | Helped |
| add chain of semantics to programming (ablation) | GPT-4o programming grounding · Sr3D | accuracy: 84.8% -> 86.1% | Helped |
| add 3D Gaussian Splatting (ablation) | Programming + chain of semantics · Nr3D | accuracy: 55.1% -> 56.3% | Helped |
| add 3D Gaussian Splatting (ablation) | Programming + chain of semantics · Sr3D | accuracy: 86.1% -> 89.6% | Helped |
| add 3D Gaussian Splatting (ablation) | Programming + chain of semantics · Sr3D View-Dep. | accuracy: 37.5% to 52.2%, an increase of 14.7% | Helped |
| replace GPT-4o with Qwen-VL (ablation) | Chain of Semantics Programming · Nr3D | accuracy: 60.8% -> 54.5% | Hurt |
Surprising findings
- Chain of semantics decreased Sr3D accuracy by 4.0% relative to dialogue-only prompting, attributed to template-based utterances.
- Programming reduced view-dependent accuracy on Sr3D compared to dialogue (drops of 12.8% and 15.6%).
Limitations
- Performance on view-dependent samples on Sr3D is lower than dialogue-based and some supervised baselines.
- Dense scenes with many same-class distractors are prone to grounding confusion.
- Relies on proprietary GPT-4o, limiting reproducibility and cost.
Experimental setup
Datasets: Sr3D Nr3D ScanNetV2
Benchmarks: Nr3D Sr3D
Metrics: grounding accuracy
Baselines: SAT LAR MVT ViL3DRel 3D-VisTA CoT3DRef ZS3DVG VLM-Grounder NS3D LARC
Builds on: 3D Gaussian Splatting ViperGPT VISPROG NS3D
Closest papers by meaning
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.91)
- Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding (CVPR 2024 · similarity 0.90)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.90)
- ReferSplat: Referring Segmentation in 3D Gaussian Splatting (ICML 2025 · similarity 0.90)
- SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding (NeurIPS 2025 · similarity 0.90)
- CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.