Research map / Video / Video-language understanding
HuMoCon: Concept Discovery for Human Motion Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HuMoCon, a motion-video LLM framework whose encoders are pre-trained via VQ-VAE concept discovery with velocity reconstruction and video-motion feature alignment, then fine-tuned on Llama3.1-8B for human motion QA.
Problem
Human motion understanding from video and motion-capture inputs lacks explicit cross-modal feature alignment, and masked autoencoder pre-training over-smooths temporal dynamics, losing high-frequency information needed for question answering.
Approach
Pre-train video and motion encoders with a VQ-VAE codebook of discovered motion concepts, add discriminative and actionable informativeness losses that reconstruct optical flow / delta motion via hypernetwork gradients, and align video and motion features frame-wise before LLM instruction tuning.
Key results
- On BABEL-QA, overall accuracy 0.711 vs 0.436 for MotionLLM (nearly 60% relative gain).
- On ActivityNet-QA, accuracy 54.2 and score 3.6, versus 53.3 and 3.5 for MotionLLM.
- Ablation: removing velocity reconstruction (Ldis & Lact) drops BABEL-QA overall accuracy to 0.637; removing alignment loss drops it to 0.675.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HuMoCon instead of MotionLLM | Video understanding · ActivityNet-QA test | accuracy: 53.3 -> 54.2 | Helped |
| use HuMoCon instead of MotionLLM | Video understanding · ActivityNet-QA test | score: 3.5 -> 3.6 | Helped |
| use HuMoCon instead of MotionLLM | Motion understanding · BABEL-QA test | Overall: 0.436 -> 0.711 | Helped |
| add masked reconstruction loss (ablation) | HuMoCon · BABEL-QA test | Overall: 0.696 -> 0.711 | Helped |
| add discriminative and actionable informativeness losses for velocity reconstruction (ablation) | HuMoCon · BABEL-QA test | Overall: 0.637 -> 0.711 | Helped |
| add feature alignment loss (ablation) | HuMoCon · BABEL-QA test | Overall: 0.675 -> 0.711 | Helped |
Limitations
- Lacks human-environment contact and object interaction understanding.
- Struggles with niche motions.
- MotionLLM's benchmark was not released, so direct comparison on it was not possible.
Experimental setup
Datasets: Motion-X HumanML3D VATEX Video-ChatGPT BABEL-QA ActivityNet-QA
Benchmarks: BABEL-QA ActivityNet-QA
Metrics: accuracy LLM-evaluated score
Baselines: 2s-AGCN MotionCLIP MotionLLM FrozenBiLM VideoChat LLaMA-Adapter Video-LLaMA Video-ChatGPT Video-LLaVA VideoChat2
Builds on: MotionLLM InfoCon VQ-VAE Video-LLaVA LART
Closest papers by meaning
- Programmatic Concept Learning for Human Motion Description and Synthesis (CVPR 2022 · similarity 0.89)
- Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition (NeurIPS 2025 · similarity 0.87)
- FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding (NeurIPS 2025 · similarity 0.87)
- HumMUSS: Human Motion Understanding using State Space Models (CVPR 2024 · similarity 0.86)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.86)
- Scaling Large Motion Models with Million-Level Human Motions (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.