Research map / Vision-language and multimodal / Multimodal LLMs

HSI-GPT2: A Dual-Granularity Large Motion Reasoning Model with Diffusion Refinement for Human-Scene Interaction

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HSI-GPT2, a unified scene-motion-language model that uses a dual-granularity motion tokenizer, an LLM+diffusion motion decoder, and CoT-based GRPO reinforcement learning for human-scene interaction understanding and generation.

Problem

Unified understanding and generation of human motion in 3D scenes lags behind specialist models; single-granularity motion codebooks, limited detokenizer decoding, and SFT-only training fail to capture motion semantics and multi-step reasoning.

Approach

A dual-codebook tokenizer (DMoTok) separating CLIP-aligned semantic tokens from fine-grained detail tokens, an LLM that plans over these tokens with a latent diffusion (MLD-style) decoder producing motions, and GRPO fine-tuning with format, motion-fidelity, and semantic-alignment rewards on a Motion Chain-of-Thought dataset generated by Qwen3-VL-235B.

Key results

• On HumanML3D text-to-motion, HSI-GPT2 reaches FID 0.139 and R-Precision Top-3 0.835, versus 0.187 FID and 0.788 for HSI-GPT.
• On HUMANISE, goal distance drops to 0.143 (vs 0.182 for HSI-GPT) and contact rate reaches 97.98%.
• On the novel evaluation set, R-Precision Top-3 is 0.624 vs 0.587 for HSI-GPT and FID is 4.892 vs 6.452.

Limitations

• CoT and perceptual evaluation rely on a closed API model (Gemini-2.5 Pro) and Qwen3-VL-235B-generated supervision.
• CoT post-training is omitted on HUMANISE because its commands are short and simple, so reasoning benefits are not shown there.

Experimental setup

Datasets: HumanML3D HUMANISE PROX AMASS Novel Evaluation Set (Afford-Motion)

Benchmarks: HumanML3D HUMANISE AMASS Novel Evaluation Set

Metrics: FID R-Precision MM Dist Diversity MModality APD Goal Dist Contact Non-collision BLEU

Baselines: HSI-GPT MotionGPT MotionGPT-2 MotionAgent Afford-Motion MoMask MMM MoGenTS MotionAnything MotionCLR Motion-R1 MotionGPT-3

Builds on: HSI-GPT MotionGPT MLD GRPO Motion-R1

Closest papers by meaning

• HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.93)
• MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities (CVPR 2025 · similarity 0.87)
• AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.87)
• UniMo: Unified Motion Generation and Understanding with Chain of Thought (AAAI 2026 · similarity 0.87)
• HMVLM:Human Motion-Vision-Language Model via MoE LoRA (NeurIPS 2025 · similarity 0.87)
• OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.86)