Research map / 3D vision / Human pose estimation
VideoMDM: Towards 3D Human Motion Generation From 2D Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a 3D text-to-motion diffusion model (VideoMDM) using only 2D keypoint supervision via reprojection against a noisy 2D-to-3D lifter teacher, with depth weighting, velocity, and representation-alignment losses.
Problem
Training generative 3D human motion priors without 3D ground truth, using only accurate 2D poses extracted from monocular video, since MoCap data is limited in diversity.
Approach
Diffuse noisy 3D lifter outputs, denoise in 3D, and supervise in 2D via depth-weighted camera reprojection (shown equivalent in expectation to 3D MSE), plus 2D-adapted velocity loss and ray-projected representation alignment; LIS-style multistep denoising for low-noise timesteps.
Key results
- On HumanML3D, Ours/MVLift with GT cameras reaches FID 0.876 vs 0.544 for 3D-supervised MDM and 1.671 for MDM/MVLift.
- On Fit3D lifting, Ours/WHAM reduces MPJPE to 111.24 mm vs 228.47 mm for WHAM and has Accel 3.16 vs 17.66 m/s^2.
- On NBA, VideoMDM is preferred over MAS in 64% of 200 pairwise human votes, though it trails MAS on FID (7.18 vs 5.38).
Surprising findings
- VideoMDM is worse than MAS on the VAE-based FID and Diversity on NBA, yet preferred by humans, which the authors attribute to unreliable VAE-based metrics.
- Naive 3D comparison in the representation-alignment loss roughly triples FID relative to the ray-projection version.
Limitations
- Best results require ground-truth cameras, which are unavailable for most in-the-wild videos; PnP estimation leaves a larger gap on Fit3D.
- Depends on a pretrained 2D-to-3D lifter as noisy teacher; fails for domains without a reasonable lifter such as non-human motion.
- Evaluated only in settings with no or minimal occlusion.
Experimental setup
Datasets: HumanML3D Fit3D NBA
Benchmarks: HumanML3D Fit3D NBA
Metrics: FID Diversity R-Precision Multimodal Distance Multimodality MPJPE PA-MPJPE PCK Acceleration error KID
Baselines: MDM (3D data) MAS MDM/MotionBERT MDM/MVLift WHAM MVLift ElePose MotionBERT
Builds on: MDM Lesson in Splats DDIM MVLift
Closest papers by meaning
- Lifting Motion to the 3D World via 2D Diffusion (CVPR 2025 · similarity 0.91)
- FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations (CVPR 2024 · similarity 0.88)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation (ICLR 2025 · similarity 0.87)
- PoseTraj: Pose-Aware Trajectory Control in Video Diffusion (CVPR 2025 · similarity 0.87)
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.