Research map / 3D vision / 3D generation
Vision-Guided Action: Enhancing 3D Human Motion Prediction with Gaze-informed Affordance in 3D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GAP3DS, an affordance-aware 3D human motion prediction model that uses gaze to identify the intended object, infer its affordance, and generate interaction poses that condition trajectory and pose decoding.
Problem
Gaze-based 3D human motion prediction methods treat gaze as a spatial cue only, which yields accurate trajectories but physically implausible or semantically inconsistent human poses in 3D scenes.
Approach
Interprets human gaze semantically: a gaze-guided affordance learner selects the gazed object via an interaction map over SoftGroup-segmented objects, extracts visual and textual affordance features with AffordanceNet, and uses a diffusion-based pose generator plus a dual-prompted decoder (interactive location for trajectory, interactive poses for pose) to produce motion.
Key results
- On GIMO, GAP3DS achieves Traj-path 575 mm, Traj-interact 623 mm, MPJPE-path 141.2 mm, MPJPE-interact 171.4 mm, outperforming SIF3D (580/644/150.2/186.1 with GAL and APG).
- Penetration error drops to 0.95% versus 3.24% for the best baseline variant (SIF3D with GAL and APG).
- Adding GAL and APG to baselines like SIF3D and MDP consistently improves their trajectory and pose metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GAL (ablation) | GAP3DS · GIMO test | Traj-path: 642 -> 575 | Helped |
| add GAL (ablation) | GAP3DS · GIMO test | MPJPE-path: 170.1 -> 141.2 | Helped |
| add APG (ablation) | GAP3DS · GIMO test | MPJPE-path: 164.6 -> 141.2 | Helped |
| use DPM instead of one basic decoder (ablation) | GAP3DS · GIMO test | Traj-path: 582 -> 575 | Helped |
| use DPM instead of one basic decoder (ablation) | GAP3DS · GIMO test | MPJPE-path: 144.2 -> 141.2 | Helped |
| include interactive location, visual, and textual affordance attributes (ablation) | GAP3DS · GIMO test | MPJPE-interact: 174.2 -> 171.4; 186.4 -> 171.4; 189.0 -> 171.4 | Helped |
| use the standard dual-prompt setup rather than single-prompt or reverse-prompt configurations (ablation) | GAP3DS · GIMO test | MPJPE: 148.3 -> 141.2; 164.1 -> 141.2; 171.2 -> 141.2 | Helped |
| generate 3 interactive poses instead of 1 (ablation) | GAP3DS · GIMO test | MPJPE-path: 146.2 -> 141.2 | Helped |
| increase scene point density from 512 to 8192 (ablation) | GAP3DS · GIMO test | Traj-P (mm): 631 -> 575 | Helped |
| increase scene point density from 8192 to 32768 (ablation) | GAP3DS · GIMO test | Traj-P (mm): 575 -> 575 | No clear effect |
| use SoftGroup instead of SPFormer (ablation) | GAP3DS · GIMO test | MPJPE: 150.2 -> 141.2 | Helped |
| add GAL and APG to SIF3D (ablation) | SIF3D · GIMO test | Traj-path: 590 -> 580 | Helped |
Surprising findings
- Reversing the prompts (affordance for pose, interactive poses for trajectory) hurts trajectory accuracy markedly (637 vs 575 mm).
- Increasing scene points from 8192 to 32768 gives negligible MPJPE gain (141.2 to 141.0) at much higher VRAM and lower speed.
Limitations
- Evaluated only on GIMO in the main text; GTA-IM results are deferred to the supplementary material.
- Depends on the quality of the SoftGroup scene parsing and AffordanceNet affordance features.
Experimental setup
Datasets: GIMO
Benchmarks: GIMO
Metrics: Traj-path Traj-interaction MPJPE-path MPJPE-interaction penetration error
Baselines: AuxFormer BiFU MDP SIF3D
Builds on: SIF3D AffordanceNet SoftGroup VPoser MDP
Closest papers by meaning
- Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance (CVPR 2024 · similarity 0.89)
- GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding (CVPR 2025 · similarity 0.89)
- Multimodal Sense-Informed Forecasting of 3D Human Motions (CVPR 2024 · similarity 0.88)
- Harmonizing Stochasticity and Determinism: Scene-responsive Diverse Human Motion Prediction (NeurIPS 2024 · similarity 0.88)
- Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding (AAAI 2025 · similarity 0.88)
- Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.