Research map / 3D vision / Human pose estimation
MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a musculoskeletal MANO hand model (MS-MANO) from MyoHand muscle data and a simulation-in-the-loop refiner (BioPR) that uses a PPO-trained muscle inverse-dynamics net to refine hand pose estimates on DexYCB and OakInk.
Problem
Hand pose tracking from video produces unnatural or physically implausible poses because existing hand models are joint-actuated and ignore musculoskeletal biomechanics, and per-frame estimates are unstable under occlusion or motion blur.
Approach
Integrate a Hill-type muscle-tendon system (from MyoHand/OpenSim) into the parametric MANO hand via a joint-centric muscle mapping, learn muscle excitations with an inverse-dynamics network trained by PPO against PD-controller torques, and refine estimated poses with an MLP that takes the predicted pose and a muscle-simulated reference pose.
Key results
- On DexYCB, gSDF + BioPR reduces MPJPE from 14.4 to 12.81 mm and Deformer + BioPR from 13.64 to 12.92 mm.
- BioPR's IDNet has about 0.1M parameters and Refine Net about 0.01M, with 9 ms added inference cost.
- Temporal smoothing and TOCH improved gSDF MPJPE by less than 0.8 mm, while PCA increased MPJPE by 7.91 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BioPR pose refinement | gSDF · DexYCB S0 | MPJPE: 14.4 -> 12.81 | Helped |
| Add BioPR pose refinement | gSDF · DexYCB S0 | AUC: 89.1 -> 89.7 | Helped |
| Add BioPR pose refinement | gSDF · DexYCB S0 | AE: 30.3 -> 29.9 | Helped |
| Add BioPR pose refinement | Deformer · DexYCB S0 | MPJPE: 13.64 -> 12.92 | Helped |
| Add BioPR pose refinement | Deformer · DexYCB S0 | AUC: 89.6 -> 90.4 | Helped |
| Add BioPR pose refinement | Deformer · DexYCB S0 | AE: 31.7 -> 30.7 | Helped |
| Use temporal smoothing or TOCH instead of BioPR (ablation) | gSDF · DexYCB | MPJPE: less than a 0.8mm improvement | Hurt |
| Use PCA pose priors instead of BioPR (ablation) | gSDF · DexYCB | MPJPE: 7.91mm increase | Hurt |
| Vary muscle insertion points (ablation) | MS-MANO and IDNet · Muscle-driven joint trajectories | IDNet convergence | Hurt |
Surprising findings
- PCA-based priors increased MPJPE by 7.91 mm, showing manual priors can hurt accuracy in this setting.
Limitations
- Failures from incorrect ground-truth annotations and persistent base-model errors, since BioPR depends on initial predictions.
- Extremely slow motion yields small excitations that make dynamics estimation difficult.
- Muscle insertion points require manual revision with human experts and are sensitive to configuration.
Experimental setup
Datasets: DexYCB OakInk MyoHand muscle-tendon data (MoBL, 2nd-Hand models via OpenSim)
Benchmarks: DexYCB OakInk
Metrics: MPJPE AUC acceleration error torque/trajectory comparison with MyoHand
Baselines: VIBE TCMR MeshGraphormer gSDF Deformer MyoHand temporal smoothing PCA TOCH
Builds on: MANO MyoHand Hill-type muscle model PPO gSDF Deformer
Closest papers by meaning
- emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation (NeurIPS 2024 · similarity 0.87)
- CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects (NeurIPS 2025 · similarity 0.84)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.83)
- Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation (NeurIPS 2024 · similarity 0.83)
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.83)
- Muscles in Time: Learning to Understand Human Motion In-Depth by Simulating Muscle Activations (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.