Research map / Video / Video segmentation and tracking
HumMUSS: Human Motion Understanding using State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HumMUSS, an attention-free spatiotemporal model built on diagonal state space models (gated DSSM blocks) for human motion understanding from 2D keypoints, matching transformer accuracy while being faster for sequential inference and robust to frame rate changes.
Problem
Transformer-based models for human motion understanding from video are slow for real-time sequential prediction, have quadratic training cost on long sequences, and fail to generalize to unseen frame rates.
Approach
Replace attention with stacked spatial and temporal bidirectional/unidirectional Gated Diagonal SSM blocks (multiplicative gating over forward and backward DSSMs), pretrained with the MotionBERT recipe; the continuous-time SSM formulation lets the discretization step adapt to new frame rates and gives O(1) per-frame stateful inference.
Key results
- On MPI-INF-3DHP causal setting, HumMUSS (f) reaches 21.0 MPJPE vs MotionBERT (f) 21.1 with 16M parameters; bidirectional HumMUSS (f) 16.3 MPJPE vs MotionBERT (f) 16.0.
- On NTU-RGB+D cross-subject, HumMUSS achieves 92.0% vs MotionBERT 93.0%; cross-view 97.4% vs 97.2%.
- Sequential inference on 243 frames is 3.8x more memory efficient and 11.1x faster than MotionBERT on an 80GB A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use causal HumMUSS instead of causal MotionBERT | 3D pose estimation · MPI-INF-3DHP | MPJPE (mm): 21.1 -> 21.0 | Helped |
| use bidirectional HumMUSS instead of MotionBERT | 3D pose estimation · MPI-INF-3DHP | MPJPE (mm): 16.0 -> 16.3 | Hurt |
| use causal HumMUSS instead of causal MotionBERT | 3D pose estimation · MPI-INF-3DHP | MPJPE (mm): 25.4 -> 24.6 | Helped |
| use causal HumMUSS instead of MotionBERT | human mesh recovery · 3DPW | MPJPE (mm): 82.3 -> 82.0 | Helped |
| use HumMUSS with HybrIK refinement instead of MotionBERT with HybrIK | human mesh recovery · 3DPW | MPJPE (mm): 70.1 -> 69.8 | Helped |
| use bidirectional HumMUSS instead of MotionBERT | human mesh recovery · 3DPW | MPJPE (mm): 76.9 -> 77.4 | Hurt |
| use bidirectional HumMUSS with HybrIK refinement instead of MotionBERT with HybrIK | human mesh recovery · 3DPW | MPJPE (mm): 68.8 -> 69.1 | Hurt |
| use HumMUSS instead of SkeleTR [C] | skeleton-based action recognition · NTU-RGB+D cross-subject | top-1 accuracy: 94.8 -> 92.0 | Hurt |
| use HumMUSS instead of SkeleTR [C] | skeleton-based action recognition · NTU-RGB+D cross-view | top-1 accuracy: 97.7 -> 97.4 | Hurt |
| use HumMUSS instead of MotionBERT for sequential inference | causal 3D pose estimation · 243 frames | inference speed: 11.1× faster | Helped |
| use HumMUSS instead of MotionBERT for sequential inference | causal 3D pose estimation · 243 frames | GPU memory usage: 3.8× memory efficient | Helped |
| evaluate at unseen or changing frame rates using HumMUSS | 3D pose estimation · MPI-INF-3DHP | MPJPE | Helped |
Surprising findings
- Transformer MotionBERT's MPJPE degrades markedly under higher sub-sampling rates while HumMUSS remains robust.
- HumMUSS converges significantly faster than MotionBERT during training for both causal and bidirectional models.
Limitations
- Mesh recovery from 2D keypoints alone remains weaker than image-based methods; gains require a hybrid with HybrIK.
- Does not beat SkeletonTR on action recognition and the authors leave task-specific head improvements to future work.
- Training dynamics explanation for faster convergence is left unexplored.
Experimental setup
Datasets: AMASS MPI-INF-3DHP PoseTrack InstaVariety 3DPW COCO NTU-RGB+D
Benchmarks: MPI-INF-3DHP 3DPW NTU-RGB+D
Metrics: MPJPE PCK AUC MPVE PA-MPJPE top-1 accuracy latency GPU memory
Baselines: MotionBERT MotionAGFormer MHFormer MixSTE PoseFormerV2 STCFormer Pose2Mesh HybrIK SkeleTR PoseConv3D
Builds on: MotionBERT S4 Diagonal State Spaces (DSS) Gated State Spaces (GSS) Hyena
Closest papers by meaning
- SimpliHuMoN: Simplifying Human Motion Prediction (NeurIPS 2026 · similarity 0.87)
- HuMoCon: Concept Discovery for Human Motion Understanding (CVPR 2025 · similarity 0.86)
- PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos (CVPR 2024 · similarity 0.86)
- SpatioTemporal Learning for Human Pose Estimation in Sparsely-Labeled Videos (AAAI 2025 · similarity 0.85)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.85)
- Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.