Research map / 3D vision / Human pose estimation
PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PhysPT, a self-supervised physics-aware Transformer encoder-decoder trained only on AMASS motion capture that refines kinematics-based monocular 3D human motion and infers contact and joint forces.
Problem
Monocular 3D human motion estimates are physically implausible (jitter, foot skating, ground penetration) because kinematics-based methods ignore physics, and existing physics-based methods need 3D-annotated videos or physics engines.
Approach
A Transformer encoder-decoder pretrained by reconstruction on mocap data, augmented with a Phys-SMPL body representation (analytic mass/inertia from SMPL shape), a continuous spring-mass contact force model, and physics-inspired losses (force, contact, Euler-Lagrange) so it can be plugged onto any kinematic estimator without fine-tuning.
Key results
- On Human3.6M, applying PhysPT to CLIFF cuts acceleration error from 15.4 to 2.5 mm/frame^2 (83.8% reduction) and foot skating from 8.3 to 2.6 mm while keeping MJE near 52.7 mm.
- On 3DOH, PhysPT reduces CLIFF's velocity error from 12.0 to 6.5 and P-MJE from 36.8 to 33.3 mm.
- On PennAction, using PhysPT motion plus forces raises human action recognition top-1 accuracy from 96.0% to 98.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PhysPT to CLIFF kinematics estimates | CLIFF · Human3.6M test | ACCL: 15.4 -> 2.5 | Helped |
| Apply PhysPT to CLIFF kinematics estimates | CLIFF · 3DOH test | VEL: 12.0 -> 6.5 | Helped |
| Use PhysPT motion and forces rather than kinematics-based joints alone | PennAction action recognition · PennAction | Top-1 accuracy: 96.0 -> 98.0 | Helped |
| Train the Transformer encoder-decoder with reconstruction loss (ablation) | PhysPT · Human3.6M test | ACCL: 15.4 -> 2.5 | Helped |
| Add force loss (ablation) | PhysPT with reconstruction loss · Human3.6M test | FS: 7.1 -> 6.5 | Helped |
| Add force loss (ablation) | PhysPT with reconstruction loss · Human3.6M test | ACCL: 2.5 -> 2.5 | No clear effect |
| Add contact loss (ablation) | PhysPT with reconstruction and force losses · Human3.6M test | FS: 6.5 -> 4.1 | Helped |
| Add contact loss (ablation) | PhysPT with reconstruction and force losses · Human3.6M test | MJE: 52.7 -> 53.0 | Hurt |
| Add Euler-Lagrange loss (ablation) | PhysPT with reconstruction, force, and contact losses · Human3.6M test | FS: 4.1 -> 2.6 | Helped |
Surprising findings
- Forces alone (without joint positions) yield 94.4% action recognition accuracy, above the prior UNIK result of 94.0%.
Limitations
- Contact model is applied only to a subset of vertices per body part for computational efficiency.
- Reconstruction accuracy slightly degrades when the contact loss is added (Table 3).
Experimental setup
Datasets: AMASS Human3.6M 3DOH PennAction
Benchmarks: Human3.6M 3DOH PennAction
Metrics: MJE P-MJE ACCL VEL FS GP Top-1 accuracy
Baselines: CLIFF VIBE PoseBert PhysCap NeurPhys SimPoE NeurMoCon D&D Huang et al. GLoT PMCE HDM
Builds on: SMPL Euler-Lagrange equations continuous contact model PoseBert
Closest papers by meaning
- Trajectory Optimization for Physics-Based Reconstruction of 3D Human Pose From Monocular Video (CVPR 2022 · similarity 0.89)
- Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos (NeurIPS 2024 · similarity 0.88)
- H-MoRe: Learning Human-centric Motion Representation for Action Analysis (CVPR 2025 · similarity 0.86)
- PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation (NeurIPS 2026 · similarity 0.86)
- Scaling Large Motion Models with Million-Level Human Motions (ICML 2025 · similarity 0.86)
- ImDy: Human Inverse Dynamics from Imitated Observations (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.