Research map / Vision-language and multimodal / Human and embodied modeling
ScanERU: Interactive 3D Visual Grounding Based on Embodied Reference Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Embodied Reference Understanding (ERU) for 3D visual grounding, builds the semi-synthetic ScanERU dataset with synthetic pointing human point clouds on ScanRefer/ScanNet, and proposes an attention-based model fusing language, gesture, and proposal features.
Problem
3D visual grounding struggles to pick out the referred object among multiple adjacent objects with similar appearance, where language alone is ambiguous.
Approach
Adds a human agent's pointing gesture, represented as a skeleton point cloud, as an extra modality alongside the scene point cloud and language, fused with transformer attention to disambiguate referents in 3D scenes.
Key results
- On the ScanERU validation set, overall Acc@0.25 reaches 69.90 and Acc@0.5 reaches 54.44, versus 49.56/37.33 for 3DJCG (SOTA baseline).
- On the multiple-object subset, ScanERU improves over the best baseline by 24.88 points at Acc@0.25 and 21.23 at Acc@0.5.
- On the real-world ScanERU test set, overall Acc@0.25 is 49.51 versus 28.56 for 3DVG-Transformer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add gesture input to the language-only model (ablation) | ScanERU · ScanERU validation set, multiple objects | Acc@0.25IoU: 41.42 -> 66.27 | Helped |
| Add gesture input to the language-only model (ablation) | ScanERU · ScanERU validation set, multiple objects | Acc@0.5IoU: 29.90 -> 52.05 | Helped |
| Add language input to the gesture-only model (ablation) | ScanERU · ScanERU validation set, unique objects | Acc@0.25IoU: 53.98 -> 85.01 | Helped |
| Add language input to the gesture-only model (ablation) | ScanERU · ScanERU validation set, overall | Acc@0.25IoU: 54.92 -> 69.90 | Helped |
| Use ScanERU instead of 3DJCG | 3D visual grounding · ScanERU validation set, overall | Acc@0.25IoU: 49.56 -> 69.90 | Helped |
| Use ScanERU instead of 3DJCG | 3D visual grounding · ScanERU validation set, overall | Acc@0.5IoU: 37.33 -> 54.44 | Helped |
| Use ScanERU instead of 3DVG-Transformer | 3D visual grounding · ScanERU test set, overall | Acc@0.25IoU: 28.56 -> 49.51 | Helped |
| Use ScanERU instead of 3DVG-Transformer | 3D visual grounding · ScanERU test set, overall | Acc@0.5IoU: 23.80 -> 42.13 | Helped |
Surprising findings
- Gesture-only ablation performs much worse on the unique subset (53.98 vs 85.01 Acc@0.25), showing gestures alone are ambiguous when several objects share a pointing direction.
Limitations
- Proposal generation does not use gestural information, so imprecise proposals can limit gesture benefits.
- Real-world test set is small and manually annotated; gestures in training are synthetic.
- Dataset covers limited scenes and gesture variety.
Experimental setup
Datasets: ScanRefer ScanNet ScanERU
Benchmarks: ScanERU validation set ScanERU test set
Metrics: Acc@0.25IoU Acc@0.5IoU
Baselines: ScanRefer TGNN InstanceRefer 3DVG-Transformer 3DJCG 3D-SPS
Builds on: 3DVG-Transformer ScanRefer Point2Skeleton PointNet++
Closest papers by meaning
- Ges3ViG : Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding (CVPR 2025 · similarity 0.92)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.90)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.89)
- 3EED: Ground Everything Everywhere in 3D (NeurIPS 2025 · similarity 0.88)
- DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding (ICLR 2025 · similarity 0.88)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.