Research map / 3D vision / 3D generation
Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage diffusion framework that predicts a scene affordance map (joint-to-scene distance field) from a 3D scene and text, then generates language-guided human motion conditioned on that map.
Problem
Generating language-guided human motion in 3D scenes is hard because jointly modeling language, scene geometry, and motion requires lots of paired data, which is scarce for language-scene-motion triplets.
Approach
Uses scene affordance maps (a normalized distance field between scene surface points and human skeleton joints, max-pooled over time) as an intermediate representation: an Affordance Diffusion Model (Perceiver-based) predicts the map from scene and text, and an Affordance-to-Motion Diffusion Model conditions motion generation on that map and text.
Key results
- On HumanML3D, the Perceiver-based method reaches FID 0.352 vs. MDM's 0.544 (Top-1 R-Precision 0.341 vs 0.319 in the first comparison block), with comparable diversity.
- On HUMANISE, goal dist. drops to 0.156 from cVAE's 0.422, and contact rises to 95.86 from 84.06.
- On the novel evaluation set, FID is 7.887 for Ours @ Enc vs 11.848 for one-stage @ Enc.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add scene affordance as an intermediate representation | HumanML3D text-to-motion generation · HumanML3D | FID: 0.489˘.025 -> 0.352˘.109 | Helped |
| add scene affordance as an intermediate representation | HumanML3D text-to-motion generation · HumanML3D | R-Precision Top 1: 0.418˘.005 -> 0.432˘.007 | Helped |
| add scene affordance as an intermediate representation | HUMANISE motion generation · HUMANISE | goal dist.: 0.185˘.014 -> 0.156˘.006 | Helped |
| add scene affordance as an intermediate representation | HUMANISE motion generation · HUMANISE | contact: 86.43˘.845 -> 96.04˘.298 | Helped |
| add scene affordance as an intermediate representation | motion generation on the novel evaluation set · Novel evaluation set | FID: 11.848˘1.634 -> 7.887˘1.189 | Helped |
| use Perceiver instead of MLP for ADM (ablation) | ADM affordance prediction · HUMANISE affordance generation | all dist.: 1.513˘.008 -> 1.086˘.007 | Helped |
| use Point Transformer instead of MLP for ADM (ablation) | ADM affordance prediction · HUMANISE affordance generation | all dist.: 1.513˘.008 -> 1.204˘.009 | Helped |
| use Perceiver instead of MLP for ADM (ablation) | AMDM motion generation · HUMANISE | goal dist.: 0.394˘.010 -> 0.156˘.006 | Helped |
| use Perceiver instead of MLP for ADM (ablation) | AMDM motion generation · HUMANISE | contact: 73.96˘.434 -> 95.86˘.323 | Helped |
| use Perceiver instead of Point Transformer for ADM (ablation) | AMDM motion generation · HUMANISE | goal dist.: 0.164˘.010 -> 0.156˘.006 | Helped |
| use Perceiver instead of Point Transformer for ADM (ablation) | AMDM motion generation · HUMANISE | contact: 94.39˘.408 -> 95.86˘.323 | Helped |
| use Perceiver instead of Point Transformer for ADM (ablation) | AMDM motion generation · HUMANISE | non-collision: 99.82˘.008 -> 99.69˘.007 | Hurt |
Surprising findings
- The MLP affordance generator lags Perceiver and Point Transformer in grounding accuracy, and Perceiver has higher collision rates than Point Transformer.
Limitations
- Diffusion models make inference slow
- Data scarcity for language-scene-motion triplets remains a hurdle
- Fails on unseen human-scene interactions and highly complex descriptions
Experimental setup
Datasets: HumanML3D HUMANISE PROX ScanNet AMASS
Benchmarks: HumanML3D HUMANISE novel evaluation set (16 scenes, 80 descriptions)
Metrics: R-Precision FID MultiModal Dist Diversity MultiModality goal dist. APD contact non-collision quality score
Baselines: Language2Pose T2M MDM cVAE one-stage diffusion model
Builds on: MDM Perceiver Motion diffusion model HUMANISE
Closest papers by meaning
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.90)
- Vision-Guided Action: Enhancing 3D Human Motion Prediction with Gaze-informed Affordance in 3D Scenes (CVPR 2025 · similarity 0.89)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.89)
- Generating Human Motion in 3D Scenes from Text Descriptions (CVPR 2024 · similarity 0.89)
- AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.88)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.