Research map / 3D vision / 3D generation
Ges3ViG : Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Imputer, an automated pipeline that inserts pointing human avatars into ScanNet scenes and builds the ImputeRefer benchmark, plus Ges3ViG, a unified model that localizes the pointing human and fuses gesture with language for 3D embodied reference understanding.
Problem
3D embodied reference understanding (3D-ERU) requires identifying a target object in a 3D scene from a language description plus a human pointing gesture, but existing data is scarce and flawed (manually placed avatars too close to targets, unrealistic language) and models assume the human's pose is
Approach
Automatically place SMPL-X-based human avatars in vacant, line-of-sight voxel regions of ScanNet scenes, regenerate referring expressions with Gemini conditioned on the pointing context, and train a model that jointly localizes the human joints and fuses early (language+gesture concatenation) and late (angular pointing bias) gesture cues with 3D scene features.
Key results
- Ges3ViG reaches 58.71% overall IoU@0.5 on ImputeRefer vs 28.84% for ScanERU (about 29.9 points higher).
- Ges3ViG beats M3DRefCLIP by 8.93% overall IoU@0.5 (70.85 vs 65.53 IoU@0.25 overall; 58.71 vs 49.78 IoU@0.5).
- Removing the human localization loss drops overall IoU@0.5 from 58.71 to 43.51.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ges3ViG instead of ScanERU | 3D-ERU grounding · ImputeRefer overall | IoU@0.5: 28.84 -> 58.71 | Helped |
| Use Ges3ViG instead of M3DRefCLIP | 3D grounding · ImputeRefer overall | IoU@0.5: 49.78 -> 58.71 | Helped |
| Use generated ImputeRefer instructions instead of original ScanRefer instructions | Ges3ViG · ImputeRefer overall | IoU@0.5: 59.69 -> 58.71 | Hurt |
| Add gestural reasoning (ablation) | Ges3ViG · ImputeRefer overall | IoU@0.5: 39.54 -> 58.71 | Helped |
| Add human localization loss (ablation) | Ges3ViG · ImputeRefer overall | IoU@0.5: 43.51 -> 58.71 | Helped |
| Add early fusion alongside late fusion (ablation) | Ges3ViG · ImputeRefer overall | IoU@0.5: 41.00 -> 58.71 | Helped |
| Add late gesture fusion to early fusion (ablation) | Ges3ViG · ImputeRefer overall | IoU@0.5: 58.05 -> 58.71 | Helped |
| Use learned late-fusion weights instead of random weights (ablation) | Ges3ViG · ImputeRefer overall | IoU@0.5: 57.7 -> 58.71 | Helped |
| Add language information to gesture-only grounding (ablation) | Ges3ViG · ImputeRefer overall | IoU@0.5: 10.18 -> 58.71 | Helped |
| Use specific language references instead of the constant “This object” (ablation) | Ges3ViG · ImputeRefer overall | IoU@0.5: 38.15 -> 58.71 | Helped |
Surprising findings
- Using original ScanRefer long descriptions gave slightly higher overall IoU@0.5 (59.69) than the shorter regenerated ImputeRefer instructions (58.71).
- Ges3ViG with only gesture information (onlyGest) reaches just 10.18% IoU@0.5, and a constant 'This object' instruction reaches 38.15%, showing language remains essential when the pointer is far away.
Limitations
- ScanERU is no longer publicly available, so only a partial test set was used for comparison.
- Scenes assume a level floor and avatars are pre-aligned to the floor, limiting applicability.
- Only 12 avatar meshes and a single VLM (Gemini) generate the referring expressions, limiting diversity.
Experimental setup
Datasets: ScanRefer ScanNetV2 SMPL-X SMPLite-X
Benchmarks: ImputeRefer
Metrics: IoU@0.25 IoU@0.5
Baselines: 3DVG-Transformer HAM 3DJCG RefMask-3D M3DRefCLIP ScanERU
Builds on: M3DRefCLIP PointGroup ScanERU Point2Skeleton PointNet
Closest papers by meaning
- ScanERU: Interactive 3D Visual Grounding Based on Embodied Reference Understanding (AAAI 2024 · similarity 0.92)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.88)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.87)
- 3EED: Ground Everything Everywhere in 3D (NeurIPS 2025 · similarity 0.87)
- GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding (CVPR 2025 · similarity 0.87)
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.