Research map / Vision-language and multimodal / Human and embodied modeling
HMVLM:Human Motion-Vision-Language Model via MoE LoRA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HMVLM, a MoE LoRA-based multimodal LLM for human motion, vision and language tasks, using a non-trainable zero expert and body-part-based pose/motion tokenizers to reduce catastrophic forgetting during instruction tuning.
Problem
Instruction tuning foundation language models with human motion tokens causes catastrophic forgetting of general linguistic knowledge, and autoregressive-compatible pose tokenization lacks spatial granularity for heterogeneous human-centric tasks.
Approach
A gating network routes task instructions over multiple LoRA experts, including a non-trainable zero expert that preserves pretrained weights; a gate loss pushes motion-unrelated prompts to the zero expert, and body-part-specific tokenizers with spatial transformers discretize poses per body part.
Key results
- On MT-Bench, the Gemma-2-2b-it based model drops only 3.34% (7.79 to 7.53) after T2M tuning, versus 87.16% for MotionAgent (7.79 to 1.00).
- On HumanML3D multi-task setting, HMVLM reaches Top-1 R-precision 0.463 and FID 0.156, below the single-task model (0.502 and 0.123) and trailing MotionGPT-2 and MoMask on most metrics.
- On 3DPW pose estimation, the single-task HMVLM obtains MPJPE 105.3 vs 163.6 for ChatPose, while HMR 2.0 remains best at 91.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoE LoRA with gating loss rather than fine-tuning without the gating loss (ablation) | HMVLM on Vicuna · MT-Bench after text-to-motion tuning | Avg: 5.90 -> 1.00 | Helped |
| Use body-part-based tokenization instead of a whole-body tokenizer (K=512) (ablation) | HMVLM text-to-motion tokenizer · HumanML3D | R-precision Top-3: 0.741 ±.001 -> 0.785 ±.002 | Helped |
| Use body-part-based tokenization instead of a whole-body tokenizer (K=512*5) (ablation) | HMVLM text-to-motion tokenizer · HumanML3D | R-precision Top-3: 0.758 ±.003 -> 0.785 ±.002 | Helped |
| Use body-part-based tokenization instead of a whole-body tokenizer (K=512) (ablation) | HMVLM motion tokenizer · HumanML3D | FID: 0.336 ±.012 -> 0.123 ±.004 | Helped |
| Use body-part-based tokenization instead of a whole-body tokenizer (K=512*5) (ablation) | HMVLM motion tokenizer · HumanML3D | FID: 0.110 ±.008 -> 0.123 ±.004 | Hurt |
| Use body-part-based tokenization instead of a whole-body tokenizer (K=512) (ablation) | HMVLM pose and motion tokenizer · HumanML3D | MSE: 1.377 -> 0.966 | Helped |
| Use body-part-based tokenization instead of a whole-body tokenizer (K=512*5) (ablation) | HMVLM pose and motion tokenizer · HumanML3D | MSE: 1.34 -> 0.966 | Helped |
| Fine-tune jointly on multiple tasks rather than only text-to-motion | HMVLM · HumanML3D | R-precision Top-1: 0.502 -> 0.463 | Hurt |
| Fine-tune jointly on multiple tasks rather than only text-to-motion | HMVLM · HumanML3D | FID: 0.123 -> 0.156 | Hurt |
| Use HMVLM's MoE LoRA and zero-expert design for T2M tuning instead of MotionAgent's tuning approach | Gemma2 foundation model · MT-Bench after tuning | Avg: 1.00 -> 7.53 | Helped |
| Use body-part tokenization and MoE LoRA in single-task pose estimation instead of ChatPose | HMVLM · 3DPW | MPJPE: 163.6 -> 105.3 | Helped |
Surprising findings
- Multi-task HMVLM underperforms its single-task variant on text-to-motion, and the paper attributes this to shared expert capacity.
- Pose estimation results still lag specialized regressors like HMR 2.0 and SPIN by a wide margin.
Limitations
- Modality connections are learned in an independent pairwise manner, limiting holistic integration.
- Domain discrepancies across datasets hinder seamless any-to-any generation.
- Video understanding can miss key poses when frame sampling is sparse, causing semantic errors.
Experimental setup
Datasets: HumanML3D KIT-ML Human3.6M 3DPW MoVid LMSYS-Chat-1M
Benchmarks: MT-Bench HumanML3D KIT-ML Human3.6M 3DPW
Metrics: MT-Bench score R-precision FID MM-Distance Diversity MPJPE PA-MPJPE MSE
Baselines: MotionGPT MotionAgent MotionGPT-2 M3GPT MoMask ReMoDiffuse ChatPose HMR 2.0 SPIN
Builds on: MoE LoRA T2M-GPT MotionGPT LLaVA
Closest papers by meaning
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.88)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.87)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.87)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.87)
- CogVLA: Cognition-Aligned Vision-Language-Action Models via Instruction-Driven Routing & Sparsification (NeurIPS 2025 · similarity 0.87)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.