Research map / Vision-language and multimodal / Human and embodied modeling
Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LARC, a neuro-symbolic 3D visual grounding concept learner regularized with LLM-distilled language constraints (symmetry, exclusivity, synonymy, sparsity), enabling naturally supervised training from only scene and QA pairs.
Problem
3D visual grounding (referring expression comprehension) typically needs dense supervision such as object-level class labels; learning from only 3D scenes and QA pairs (naturally supervised) causes prior methods to underperform.
Approach
Regularize structured neuro-symbolic concept learner intermediate representations with language-derived constraints (symmetry, exclusivity, synonymy, sparsity) whose concept sets are distilled from an LLM (GPT-3.5), plus synonym-substitution data augmentation and LLM-based compositional execution of unseen concepts.
Key results
- LARC reaches 36.6% test accuracy on ReferIt3D SR3D in the naturally supervised setting, up from 27.6% for NS3D (+9 points).
- LARC achieves 37.4% ternary accuracy and 32.6% antonym accuracy on zero-shot composition, versus 19.1% and 16.0% for NS3D.
- On ScanRefer transfer, LARC scores 32.9% versus 22.4% for NS3D and 18.1% for BUTD-DETR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add language-based regularization to NS3D | NS3D · SR3D test | accuracy: 27.6 -> 36.6 | Helped |
| use language-based composition for unseen concepts | LARC · SR3D unseen ternary relations | accuracy: 19.1 -> 37.4 | Helped |
| use language-based composition for unseen concepts | LARC · SR3D unseen antonyms | accuracy: 22.0 -> 32.6 | Helped |
| train with LARC rather than the strongest baseline | 3D referring expression comprehension · SR3D with 10% training data | accuracy: 22.3 -> 29.1 | Helped |
| transfer learned concepts using LARC | LARC · ScanRefer transfer subset | accuracy: 18.1 -> 32.9 | Helped |
| add symmetry regularization (ablation) | LARC · SR3D test | accuracy: 34.9 -> 36.6 | Helped |
| add exclusivity regularization (ablation) | LARC · SR3D test | accuracy: 35.1 -> 36.6 | Helped |
| add sparsity regularization (ablation) | LARC · SR3D test | accuracy: 35.8 -> 36.6 | Helped |
| add synonymity augmentation (ablation) | LARC · SR3D test | accuracy: 33.4 -> 36.6 | Helped |
| add object-level classification supervision (ablation) | LARC · SR3D test | accuracy: 36.6 -> 38.2 | Helped |
Surprising findings
- Synonymity regularization has the largest ablation effect (removing it drops accuracy to 33.4%), while sparsity has the smallest effect.
- Classification supervision gives smaller LARC gains (+6.6 points over NS3D) than in the naturally supervised setting, suggesting regularization matters most when dense labels are absent.
Limitations
- Accuracy in the naturally supervised setting (36.6%) remains only comparable to end-to-end methods, not clearly superior.
- Composition rules are applied only to a chosen subset of relations and require manual or LLM-derived rule lookup.
- Evaluated mainly on ReferIt3D SR3D with a limited antonym test set of 50 examples.
Experimental setup
Datasets: ReferIt3D (SR3D) ScanNet ScanRefer
Benchmarks: ReferIt3D SR3D ScanRefer transfer subset
Metrics: accuracy symmetric subset accuracy exclusive subset accuracy ternary accuracy antonym accuracy data efficiency accuracy
Baselines: NS3D BUTD-DETR MVT LAR TransRefer3D LanguageRefer
Builds on: NS3D Neuro-Symbolic Concept Learner VoteNet PointNet++
Closest papers by meaning
- From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs (ICML 2025 · similarity 0.90)
- Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.89)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.89)
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.89)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.89)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.