Research map / Vision-language and multimodal / Human and embodied modeling
HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HSI-GPT, an LLM-based scene-motion-language model that tokenizes SMPL-X motions with a VQ-VAE, embeds 3D scenes and affordance maps via a Q-Former-style aggregator, and instruction-tunes LLaMA 3-8B with LoRA for text- and pose-conditioned human-scene interaction generation, captioning, and motion completion.
Problem
Language-conditioned human-scene interaction (HSI) motion synthesis is limited by task-specific cVAE/diffusion models, restricted control modalities, and weak alignment among 3D scene, motion, and language, leading to descriptive-unfaithful, physically implausible motions.
Approach
Treat motions as discrete "body language" tokens in a unified motion-language vocabulary of an LLM, align a learned scene/affordance query aggregator (MIA) to the LLM embedding space, and unify HSI generation, captioning, and completion as instruction-following next-token prediction.
Key results
- On HumanML3D, HSI-GPT reports R-Precision Top1 0.495 and FID 0.187 and MM Dist 3.058, outperforming LLM-based text-to-motion baselines in those metrics.
- On HUMANISE text-conditioned HSI generation, HSI-GPT (LLaMA 3-8B) achieves Goal Dist 0.182, Contact 92.31, Non-collision 99.82, competitive with specialist cVAE/Afford-Motion baselines.
- For HSI captioning on augmented HumanML3D, HSI-GPT reaches R-Precision Top1 0.551 and CIDEr 30.8, the highest among compared captioning methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HSI-GPT instead of Afford-Motion 2-stage@En | Text-conditioned HSI generation · HUMANISE | Goal Dist: 0.156±.006 -> 0.182±.008 | Hurt |
| Use HSI-GPT instead of Afford-Motion 2-stage@De | Text-conditioned HSI generation · HUMANISE | Contact: 96.04±.298 -> 92.31±.478 | Hurt |
| Use HSI-GPT instead of cVAE | Text-conditioned HSI generation · HUMANISE | N-collision: 99.77±.004 -> 99.82±.005 | Helped |
| Add initial motion token control (ablation) | HSI-GPT · HumanML3D | FID: 0.187 -> 0.185 | Helped |
| Add initial motion token control (ablation) | HSI-GPT · HUMANISE | Contact: 92.31 -> 92.67 | Helped |
| Add last motion token control (ablation) | HSI-GPT · HumanML3D | FID: 0.187 -> 0.234 | Hurt |
| Add last motion token control (ablation) | HSI-GPT · HUMANISE | Contact: 92.31 -> 92.45 | Helped |
| Add random motion token controls (ablation) | HSI-GPT · HumanML3D | FID: 0.187 -> 0.180 | Helped |
| Add random motion token controls (ablation) | HSI-GPT · HUMANISE | Contact: 92.31 -> 92.86 | Helped |
| Use LLaMA 3-8B instead of smaller foundational LLMs (ablation) | HSI-GPT · HUMANISE | Goal Dist: 0.332±.019 -> 0.182±.008 | Helped |
Surprising findings
- Text+Last pose conditioning degraded FID (0.234) on HumanML3D relative to text-only, with the authors attributing it to overly goal-oriented motions.
Limitations
- Reduced motion diversity (APD) under strict key-frame pose control.
- Text+Last pose conditioning overemphasizes goal-orientation, hurting smoothness and diversity.
Experimental setup
Datasets: HumanML3D HUMANISE PROX
Benchmarks: HumanML3D HUMANISE Novel Evaluation Set
Metrics: FID R-Precision MM Dist Diversity MultiModality Goal Dist APD Contact Non-collision BLEU
Baselines: cVAE (HUMANISE) Afford-Motion MotionGPT MotionLLM T2M-GPT MDM MotionDiffuse MLD TM2T T2M AttT2M
Builds on: MotionGPT LLaVA Afford-Motion T2M-GPT Q-Former
Closest papers by meaning
- HSI-GPT2: A Dual-Granularity Large Motion Reasoning Model with Diffusion Refinement for Human-Scene Interaction (CVPR 2026 · similarity 0.93)
- Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance (CVPR 2024 · similarity 0.90)
- Unified Human-Scene Interaction via Prompted Chain-of-Contacts (ICLR 2024 · similarity 0.89)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.89)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.88)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.