Research map / 3D vision / 3D generation
Generating Human Motion in 3D Scenes from Text Descriptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates text-conditioned human motions in 3D indoor scenes by using ChatGPT to ground the target object from a scene graph, then running two diffusion models over object-centric volumetric sensors to produce trajectories and poses.
Problem
Generating human motions that follow text descriptions of human-scene interactions in 3D indoor scenes requires spatial reasoning to locate the target object and motions coherent with cluttered point-cloud scenes.
Approach
Decompose the task into LLM-based language grounding of the target object (ChatGPT over a text-converted scene graph with two prompts) and object-centric diffusion motion generation using volumetric environment, target, and trajectory sensors instead of full point clouds.
Key results
- On HUMANISE, the method reduces goal distance to 0.384 (vs 0.995 for HUMANISE) and raises perceptual scene score to 3.54 (vs 2.81 for the best baseline quality score comparison).
- ChatGPT-based grounding reaches 75.6% acc (IoU 0.25) with predicted detection vs 62.5% for BUTD-DETR; Mistral 7B reaches 72.4%.
- Action recognition accuracy of 97.1% for generated motions, close to real data at 99.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use two-stage ChatGPT target-object localization and object-centric motion generation instead of HUMANISE | text-and-scene human motion generation · HUMANISE test | goal dist.: 0.995 -> 0.384 | Helped |
| use two-stage ChatGPT target-object localization and object-centric motion generation instead of HUMANISE | text-and-scene human motion generation · HUMANISE test | scene score: 2.33 -> 3.54 | Helped |
| use two-stage ChatGPT target-object localization and object-centric motion generation instead of HUMANISE | text-and-scene human motion generation · HUMANISE test | accuracy: 89.7% -> 97.1% | Helped |
| use two-stage ChatGPT target-object localization and object-centric motion generation instead of HUMANISE | text-and-scene human motion generation · HUMANISE test | FID: 1.11 -> 0.12 | Helped |
| use two-stage ChatGPT target-object localization and object-centric motion generation instead of HUMANISE | text-and-scene human motion generation · HUMANISE test | quality score: 2.81 -> 3.50 | Helped |
| add target-object localization (ablation) | proposed motion generation method · HUMANISE test | goal dist.: 0.592 -> 0.384 | Helped |
| add target-object localization (ablation) | proposed motion generation method · HUMANISE test | accuracy: 74.5% -> 97.1% | Helped |
| add object-centric representation (ablation) | proposed motion generation method · HUMANISE test | FID: 1.01 -> 0.12 | Helped |
| generate trajectories and motions in two stages rather than together (ablation) | proposed motion generation method · HUMANISE test | goal dist.: 0.385 -> 0.384 | No clear effect |
| generate trajectories and motions in two stages rather than together (ablation) | proposed motion generation method · HUMANISE test | FID: 0.14 -> 0.12 | No clear effect |
| replace cVAE with diffusion models (ablation) | proposed motion generation method · HUMANISE test | accuracy: 40.0% -> 97.1% | Helped |
| replace cVAE with diffusion models (ablation) | proposed motion generation method · HUMANISE test | FID: 3.61 -> 0.12 | Helped |
Surprising findings
- Open-source Mistral 7B nearly matches ChatGPT for target grounding, while LLaMA 2 7B performs much worse (40.3%).
Limitations
- Only static scenes are handled; moving objects are not considered.
- Motions are short (60% around 1-3 s) and texts mostly follow the Sr3D template.
- ChatGPT fails when the object detector misses objects.
Experimental setup
Datasets: HUMANISE AMASS PROX
Benchmarks: HUMANISE PROX
Metrics: goal dist. scene score action recognition accuracy diversity multimodality FID quality score grounding acc@0.25 center dist.
Baselines: MDM GMD GMD with HUMANISE centers HUMANISE BUTD-DETR LLaMA 2 7B Mistral 7B
Builds on: MDM GMD HUMANISE NSM Sr3D
Closest papers by meaning
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.90)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.89)
- Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance (CVPR 2024 · similarity 0.89)
- Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs (ICLR 2025 · similarity 0.88)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.88)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.