Research map / Video / Video generation
FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FinePhys generates fine-grained gymnastics action videos by lifting online-estimated 2D poses to 3D via in-context learning, refining them with a Euler-Lagrange physics module, and using the fused skeletons as heatmap guidance for an AnimateDiff/SD-based video diffusion model.
Problem
Generating physically plausible fine-grained human action videos (e.g., gymnastics with large body deformations and rapid temporal changes) is unsolved for current video diffusion models, which produce limb distortions and temporal inconsistencies.
Approach
Combines 2D-to-3D lifting by in-context learning (with mean 3D pose priors) and a PhysNet module that bidirectionally re-estimates joint states using learned Euler-Lagrange dynamics parameters; the fused 3D skeletons are projected to 2D heatmaps that guide a 3D-UNet via LoRA-tuned diffusion.
Key results
- FinePhys reaches a user-study text alignment MOS of 4.13 versus 3.17 for the best baseline (finetuned AnimateDiff) on FX-JUMP/FX-TURN.
- FVD drops to 484.49 for FinePhys versus 506.26 for finetuned Follow-Your-Pose and 522.14 for finetuned AnimateDiff.
- On FineGym 2D pose evaluation, averaging data-driven and physics-predicted poses reduces MPJPE from 0.229 (S_dd) to 0.178, against 0.918 for online detection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FinePhys skeletal guidance and physics-based pose refinement | video generation · FX-JUMP and FX-TURN | CLIP-SIM* Text: 0.709 -> 0.826 | Helped |
| Add FinePhys skeletal guidance and physics-based pose refinement | video generation · FX-JUMP and FX-TURN | user study Text: 3.20 -> 4.13 | Helped |
| Add FinePhys skeletal guidance and physics-based pose refinement | video generation · FX-JUMP and FX-TURN | FVD: 529.38 -> 484.49 | Helped |
| Fuse data-driven and physically predicted 3D poses (ablation) | FinePhys pose estimation · FineGym | MPJPE: 0.229 -> 0.178 | Helped |
| Fuse data-driven and physically predicted 3D poses (ablation) | FinePhys pose estimation · FineGym | N-MPJPE: 0.215 -> 0.147 | Helped |
| Fuse data-driven and physically predicted 3D poses (ablation) | FinePhys pose estimation · FineGym | MPVPE: 0.108 -> 0.094 | Helped |
| Replace PhysNet with an MLP (ablation) | FinePhys pose estimation · FineGym | MPJPE: 0.178 -> 0.243 | Hurt |
| Replace PhysNet with an MLP (ablation) | FinePhys pose estimation · FineGym | MPVPE: 0.094 -> 0.140 | Hurt |
| Use only physically predicted poses rather than fusing with data-driven poses (ablation) | FinePhys pose estimation · FineGym | MPJPE: 0.178 -> 0.237 | Hurt |
Surprising findings
- The original CLIP-SIM metric gives the highest smoothness score to T2I-Zero outputs that are entirely irrelevant to the action, so the authors introduce CLIP-SIM*.
- Replacing PhysNet with a simple MLP markedly worsens pose guidance, showing the physics module matters beyond the lifting network alone.
Limitations
- Fails to generate complex salto actions involving simultaneous rotations and rapid aerial turns, which are excluded from the quantitative table.
- Evaluated on a small fine-tuning set of 35 classes (350 videos) from FineGym floor exercise.
- Focus on detailed frame generation may divert attention from deeper physics integration, as the authors acknowledge.
Experimental setup
Datasets: FineGym HumanArt Human3.6M AMASS
Benchmarks: FineGym FX-JUMP FineGym FX-TURN
Metrics: PickScore CLIP-SIM CLIP-SIM* FVD user study MOS MPJPE N-MPJPE MPVPE
Baselines: Control-A-Video VideoCrafter1 VideoCrafter2 Text2Video-Zero Latte Follow-Your-Pose AnimateDiff
Builds on: AnimateDiff Stable Diffusion v1.5 Follow-Your-Pose PhysDiff PhysPT
Closest papers by meaning
- FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models (CVPR 2024 · similarity 0.86)
- MultiPhys: Multi-Person Physics-aware 3D Motion Estimation (CVPR 2024 · similarity 0.85)
- PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation (NeurIPS 2026 · similarity 0.85)
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.85)
- Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos (NeurIPS 2024 · similarity 0.85)
- FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.