Research map / Video / Video-language understanding
Human Motion Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLaMo, a multimodal LLM framework that feeds raw human motion and video directly (not tokenized) into the LLM via a Motion-Video feature enhancer and a text-guided Cross Talker module for motion instruction tuning.
Problem
Existing motion-LLMs (MotionGPT, MotionLLM) convert motion into discrete tokens or text, losing fine-grained spatial-temporal detail, and treat video and motion as isolated modalities.
Approach
Keep motion as a native modality: enhance motion features with video context via cross-attention, then use a text-guided Cross Talker to select top-K viewpoint frames with adaptive local/global receptive-field aggregation and bidirectional motion-text fusion before feeding a language model.
Key results
- On MoVid-Bench motion, LLaMo reaches 55.32 overall accuracy vs 49.50 for MotionLLM and 36.86 for MotionGPT.
- On BABEL-QA, LLaMo achieves overall score 0.458, above MotionLLM's 0.436.
- On profession-swing, LLaMo achieves overall accuracy 24.80 vs 16.53 for MotionLLM; on Mo-RepCount it trails EScounts and PoseRAC on OBO and MAE but has best OBZ (0.222).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaMo's native motion representation and motion-text alignment instead of MotionLLM | MoVid-Bench-Motion · MoVid-Bench-Motion | All accuracy: 49.50 -> 55.32 | Helped |
| Use LLaMo's native motion representation and motion-text alignment instead of MotionLLM | MoVid-Bench-Motion · MoVid-Bench-Motion | All score: 3.49 -> 3.67 | Helped |
| Use LLaMo for video understanding instead of MotionLLM | MoVid-Bench-Video · MoVid-Bench-Video | All accuracy: 49.00 -> 52.33 | Helped |
| Use LLaMo for video understanding instead of MotionLLM | MoVid-Bench-Video · MoVid-Bench-Video | Body-part accuracy: 34.13 -> 33.83 | Hurt |
| Use LLaMo for video understanding instead of MotionLLM | MoVid-Bench-Video · MoVid-Bench-Video | All score: 2.97 -> 3.10 | Helped |
| Use LLaMo instead of MotionLLM | BABEL-QA · BABEL-QA | Overall score: 0.436 -> 0.458 | Helped |
| Use LLaMo instead of MotionLLM | profession-swing · profession-swing | All accuracy: 16.53 -> 24.80 | Helped |
| Use LLaMo instead of MotionLLM | profession-swing · profession-swing | All score: 1.57 -> 2.48 | Helped |
| Use LLaMo instead of EScounts | Mo-RepCount · Mo-RepCount | OBO: 0.397 -> 0.389 | Hurt |
| Use LLaMo instead of EScounts | Mo-RepCount · Mo-RepCount | OBZ: 0.198 -> 0.222 | Helped |
Surprising findings
- On Mo-RepCount, LLaMo does not outperform specialized counting models (EScounts, PoseRAC) on most metrics.
- On MoVid-Bench video, LLaMo slightly trails MotionLLM on body-part awareness.
Limitations
- Mo-RepCount results do not beat specialized counting models on most metrics.
- Swing dataset is proprietary-style custom data with GPT-4-based scoring, making comparisons hard to reproduce.
- Base LLM and training hyperparameters are not specified in the main text.
Experimental setup
Datasets: MoVid Swing HumanML3D KIT-ML Mo-RepCount RepCount BABEL-QA MoVid-Bench
Benchmarks: MoVid-Bench BABEL-QA Mo-RepCount profession-swing
Metrics: accuracy score OBO MAE OBZ RMSE overall score
Baselines: GPT-3.5 MotionGPT MotionLLM Video-LLaVA MotionCLIP EScounts PoseRAC TransRAC RepNet
Builds on: MotionLLM MotionGPT Video-LLaVA
Closest papers by meaning
- MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators (AAAI 2024 · similarity 0.87)
- AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond (CVPR 2024 · similarity 0.87)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.87)
- HMVLM:Human Motion-Vision-Language Model via MoE LoRA (NeurIPS 2025 · similarity 0.86)
- FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding (NeurIPS 2025 · similarity 0.86)
- MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.