Research map / Vision-language and multimodal / Human and embodied modeling
Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3DLFVG, a language-free training framework for 3D visual grounding that uses multi-view image CLIP features as pseudo-language features, plus relation injection modules (NRM and CRC) to model 2D-3D spatial relations.
Problem
3D visual grounding normally needs dense per-object text descriptions, which are costly to collect; existing language-free 2D methods do not transfer well to 3D because of weak 3D-language alignment and complex 3D spatial relations.
Approach
Generate pseudo-language features from CLIP visual embeddings of masked multi-view image crops (local plus global context, with noise), then inject neighboring-object relations in 2D (NRM) and enforce 2D-3D relational consistency (CRC) so that text can be substituted at inference.
Key results
- Overall ScanRefer Acc@0.25 30.53 and Acc@0.5 23.61 under language-free training, versus 26.90/16.96 for re-implemented Pseudo-Q.
- Nr3D overall Acc@0.25 18.28 and Acc@0.5 13.32; Sr3D overall Acc@0.25 19.25 and Acc@0.5 14.99, both best among language-free methods reported.
- Ablation: adding NRM and CRC on top of pseudo-language features raises ScanRefer overall Acc@0.25 from 26.88 to 30.53.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3DLFVG instead of Zero-shot-RIS | 3D visual grounding · ScanRefer val, Overall | Acc@0.25: 26.67 -> 30.53 | Helped |
| use 3DLFVG instead of Zero-shot-RIS | 3D visual grounding · Nr3D val, Overall | Acc@0.25: 16.99 -> 18.28 | Helped |
| use 3DLFVG instead of Zero-shot-RIS | 3D visual grounding · Sr3D val, Overall | Acc@0.25: 17.43 -> 19.25 | Helped |
| add random noise to pseudo-language features (ablation) | 3DLFVG · ScanRefer val, Overall | Acc@0.25: 26.88 -> 28.27 | Helped |
| add random noise to pseudo-language features (ablation) | 3DLFVG · ScanRefer val, Overall | Acc@0.5: 22.30 -> 20.62 | Hurt |
| add Neighboring Relation-aware Modeling (NRM) (ablation) | 3DLFVG · ScanRefer val, Overall | Acc@0.25: 26.88 -> 28.88 | Helped |
| add Cross-modality Relation Consistency (CRC) after NRM (ablation) | 3DLFVG · ScanRefer val, Overall | Acc@0.25: 28.88 -> 30.35 | Helped |
| set NRM neighboring-object count k to 2 rather than 0 (ablation) | 3DLFVG · ScanRefer val, Overall | Acc@0.25: 29.22 -> 29.58 | Helped |
| set NRM neighboring-object count k to 4 rather than 2 (ablation) | 3DLFVG · ScanRefer val, Overall | Acc@0.25: 29.58 -> 30.53 | Helped |
| increase NRM neighboring-object count from k=4 to k=6 (ablation) | 3DLFVG · ScanRefer val, Overall | Acc@0.25: 30.53 -> 30.12 | Hurt |
Surprising findings
- Random box selection reaches 3.97 Acc@0.25 on ScanRefer, and the method is reported as 'nearly reaching the upper limit' on the Unique subset (65.80 Acc@0.25).
Limitations
- Relies on pre-trained 2D mask generator and mask quality; mask generator produces incomplete objects, handled only by selecting masks matching target boxes.
- Performance on ScanRefer Acc@0.5 remains well below fully supervised methods (23.61 vs 36.43 for 3DVG-Transformer).
Experimental setup
Datasets: ScanNet ScanRefer Nr3D Sr3D
Benchmarks: ScanRefer Nr3D Sr3D
Metrics: Acc@0.25 IoU Acc@0.5 IoU
Baselines: Random OpenScene LLM-Grounder Pseudo-Q Zero-shot-RIS ScanRefer 3DVG-Transformer
Builds on: 3DVG-Transformer Pseudo-Q Zero-shot-RIS CLIP
Closest papers by meaning
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.92)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.91)
- DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding (ICLR 2025 · similarity 0.91)
- From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs (ICML 2025 · similarity 0.90)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.90)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.