Research map / 3D vision / Human pose estimation
Lifting Motion to the 3D World via 2D Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lifts single-view 2D pose sequences to global 3D motion (joint rotations and root trajectories) without any 3D training data, using a multi-stage pipeline of line-conditioned and multi-view 2D motion diffusion models.
Problem
Estimating global 3D motion (joint rotations and root trajectories in world coordinates) from 2D pose sequences for out-of-distribution motions where 3D ground truth is unavailable, such as athletic movements, animal motion, and human-object interactions.
Approach
Progressively establish multi-view consistency for 2D pose sequences: a line-conditioned (epipolar) 2D motion diffusion model, SDS-based multi-view 2D optimization, a synthetic strictly consistent multi-view dataset from 3D fitting and reprojection, and finally a cross-view-attention diffusion model that generates consistent multi-view 2D sequences in one pass.
Key results
- On AIST++, MVLift achieves MPJPE 110.7 and Troot 67.6, outperforming non-3D-supervised baselines and MotionBERT (MPJPE 134.0) and approaching WHAM on joint error while greatly improving root translation (Troot 164.3 for WHAM).
- On CatPlay, MVLift reaches J2D 57.0 and JC2D 39.4 versus SMPLify's 58.8 and 42.0 and MAS's 180.6 and 59.7 (2D reprojection metrics).
- In a human perceptual study on AMT, participants preferred MVLift over all baselines across AIST++, Steezy, and NicoleMove, and 39% preferred it over ground truth on AIST++.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the final learned multi-view 2D diffusion model instead of Stage 1-generated multi-view sequences (ablation) | MVLift · AIST++ | MPJPE: 135.2 -> 110.7 | Helped |
| use the final learned multi-view 2D diffusion model instead of Stage 2-optimized sequences (ablation) | MVLift · AIST++ | MPJPE: 127.4 -> 110.7 | Helped |
| optimize 2D multi-view sequences rather than optimize 3D motion directly with SDS on reprojections (ablation) | MVLift Stage 2 · AIST++ | MPJPE: 137.3 -> 110.7 | Helped |
| use line-conditioned rather than unconditional 2D motion diffusion for SDS (ablation) | MVLift · AIST++ | MPJPE: 230.4 -> 110.7 | Helped |
Surprising findings
- Using an unconditional 2D motion diffusion model for SDS (without epipolar line conditioning) fails to produce plausible 3D motions, with very large root and joint errors.
- WHAM, trained with 3D supervision, produces large root trajectory errors on AIST++ (Troot 164.3) despite competitive joint errors.
Limitations
- Relies on 2D pose estimates from off-the-shelf detectors (ViTPose, etc.), so 2D estimation errors propagate.
- Multi-view optimization does not guarantee perfect view consistency, motivating the synthetic-data stage.
- Joint and root errors remain comparable to rather than clearly better than WHAM in 3D joint position metrics.
Experimental setup
Datasets: AIST++ Steezy NicoleMove CatPlay OMOMO AMASS
Benchmarks: AIST++ Steezy NicoleMove CatPlay OMOMO
Metrics: Troot MPJPE PA-MPJPE J2D JC2D FID TO_root O-MPJPE
Baselines: MotionBERT WHAM ElePose SMPLify MAS
Builds on: DreamFusion score distillation sampling MAS ElePose SMPLify VPoser
Closest papers by meaning
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.91)
- MAS: Multi-view Ancestral Sampling for 3D Motion Generation Using 2D Diffusion (CVPR 2024 · similarity 0.89)
- Lifting by Image – Leveraging Image Cues for Accurate 3D Human Pose Estimation (AAAI 2024 · similarity 0.88)
- PoseTraj: Pose-Aware Trajectory Control in Video Diffusion (CVPR 2025 · similarity 0.88)
- PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (NeurIPS 2026 · similarity 0.88)
- PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor Space (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.