Research map / 3D vision / Human pose estimation
Muscles in Time: Learning to Understand Human Motion In-Depth by Simulating Muscle Activations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MinT, a ~10-hour synthetic muscle-activation dataset from AMASS mocap by running OpenSim/OpenSimAD muscle simulations, and benchmarks five sequence models for pose-to-muscle-activation regression.
Problem
Ground-truth muscle activation data (EMG/sEMG) is scarce, small, and costly to collect, limiting neural models that link human motion to muscle activity.
Approach
Generate large-scale synthetic muscle activation labels by running established OpenSim musculoskeletal models (lower body by Lai et al., thoracolumbar by Bruno et al.) with trajectory optimization and static optimization on AMASS mocap sequences, then benchmark motion-to-activation regression networks on the result.
Key results
- The 16-layer Transformer achieves the best results across all metrics, e.g., lower-body overall RMSE 0.048, SMAPE 45.1, PCC 0.54, versus LSTM 0.052/57.8/0.48.
- Upper-body overall PCC rises to 0.55 for the Transformer versus 0.32 for VQ-VAE, while RMSE differences across models are marginal.
- Fine-tuning only the first and last transformer block on Muscles in Action gives RMSE nearly equal to full fine-tuning (15.15 vs 15.11) but lower PCC (0.21 vs 0.27).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the Transformer instead of Mamba2 | MinT lower-body muscle activation prediction · MinT lower body, all motions | RMSE: 0.051 -> 0.048 | Helped |
| use the Transformer instead of Mamba2 | MinT lower-body muscle activation prediction · MinT lower body, all motions | SMAPE: 55.4 -> 45.1 | Helped |
| use the Transformer instead of Mamba2 | MinT lower-body muscle activation prediction · MinT lower body, all motions | PCC: 0.49 -> 0.54 | Helped |
| use the Transformer instead of Mamba2 | MinT upper-body muscle activation prediction · MinT upper body, all motions | RMSE: 0.034 -> 0.033 | Helped |
| use the Transformer instead of Mamba2 | MinT upper-body muscle activation prediction · MinT upper body, all motions | SMAPE: 112.2 -> 107.7 | Helped |
| use the Transformer instead of Mamba2 | MinT upper-body muscle activation prediction · MinT upper body, all motions | PCC: 0.50 -> 0.55 | Helped |
| limit fine-tuning to the first and last transformer layer instead of full fine-tuning (ablation) | MinT 16-layer Transformer on Muscles in Action · Muscles in Action, overall | RMSE: 15.11 -> 15.15 | No clear effect |
| limit fine-tuning to the first and last transformer layer instead of full fine-tuning (ablation) | MinT 16-layer Transformer on Muscles in Action · Muscles in Action, overall | PCC: 0.27 -> 0.21 | Hurt |
| limit fine-tuning to the first and last transformer layer instead of full fine-tuning (ablation) | MinT 16-layer Transformer on Muscles in Action · Muscles in Action, overall | SMAPE: 37.0 -> 41.6 | Hurt |
Surprising findings
- RMSE differences between architectures are marginal while PCC and SMAPE show clear gains, because many muscles are rarely activated and RMSE is scale-sensitive.
- Full-muscle-set evaluation on the 16-layer transformer shows SMAPE values above 90 for the upper body, indicating high percentage error on near-zero activations.
Limitations
- Synthetic-to-real domain gap; no validation against real EMG data is provided.
- Simulations restricted to foot-ground contact motions; object interactions mostly excluded, and some non-converging simulations were discarded, shifting the category distribution.
- Uses standard (non-subject-specific) body model parameters and semi-automatic marker adjustments.
Experimental setup
Datasets: AMASS KIT Whole-Body Human Motion Database BMLmovi BMLrub EyesJapan TotalCapture BABEL Muscles in Action
Benchmarks: MinT
Metrics: RMSE PCC SMAPE
Baselines: VQ-VAE FConv LSTM Mamba2 Transformer
Builds on: AMASS OpenSim OpenSimAD BABEL Falisse et al. trajectory optimization
Closest papers by meaning
- ImDy: Human Inverse Dynamics from Imitated Observations (ICLR 2025 · similarity 0.84)
- MyoChallenge 2025: A New Benchmark for Human Athletic Intelligence (NeurIPS 2026 · similarity 0.84)
- VidSole: A Multimodal Dataset for Joint Kinetics Quantification and Disease Detection with Deep Learning (AAAI 2025 · similarity 0.83)
- Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos (NeurIPS 2024 · similarity 0.83)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.83)
- MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.