Research map / 3D vision / 3D generation
MAS: Multi-view Ancestral Sampling for 3D Motion Generation Using 2D Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates 3D human/horse/ball-gymnastics motions by running ancestral sampling of a 2D motion diffusion model over multiple views, with triangulation and reprojection at each denoising step and 3D-consistent noise.
Problem
3D motion generation is limited by scarce and costly 3D motion capture data, while abundant in-the-wild videos only yield 2D pose data.
Approach
Simultaneously denoise V 2D views of the same motion with a 2D motion diffusion model, triangulating the views into one 3D motion and reprojecting at every denoising step, while sampling noise in 3D and projecting it orthographically to each view to keep the views consistent.
Key results
- MAS achieves FID 5.38 on NBA projections versus 10.76 for ElePose and 30.22 for MotionBERT (ground truth 1.05).
- Removing 3D noise causes mode collapse, with recall dropping to 0.01 and FID rising to 17.40.
- In a user study with 22 participants, MAS was preferred over MotionBERT, ElePose, and the DreamFusion adaptation on precision, quality, and diversity, with 83-100% preference rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MAS instead of ElePose | NBA pose generation · NBA (All views) | FID: 10.76±.45 -> 5.38±.06 | Helped |
| use MAS instead of MotionBERT | NBA pose generation · NBA (All views) | FID: 30.22±.26 -> 5.38±.06 | Helped |
| use 2 views (120°) instead of the standard MAS view setup (ablation) | MAS · NBA | FID: 5.38±.06 -> 6.87±.14 | Hurt |
| use 2 views (120°) instead of the standard MAS view setup (ablation) | MAS · NBA | Recall: 0.60±.01 -> 0.80±.01 | Helped |
| add multiview-consistent 3D noise (ablation) | MAS · NBA | FID: 17.40±.12 -> 5.38±.06 | Helped |
| add multiview-consistent 3D noise (ablation) | MAS · NBA | Recall: 0.01±2.6e−03 -> 0.60±.01 | Helped |
| use MAS instead of the DreamFusion adaptation | NBA motion generation · NBA | FID: 66.38±1.24 -> 5.38±.06 | Helped |
Surprising findings
- The DreamFusion adaptation reaches FID 66.38 and recall 0.17, performing far worse than MAS on the same diffusion model.
Limitations
- Characters sometimes fold into themselves when changing direction and change scale during motion.
- Cannot predict global position or apply textual control, inheriting limits of the 2D data.
- Requires 2D data capturing a variety of views of similar motions.
Experimental setup
Datasets: NBA videos (newly collected) Horse jumping contests (newly collected) Rhythmic ball gymnastics (newly collected) Human3.6M
Benchmarks: NBA 2D projected motions Human3.6M projected to random 2D cameras
Metrics: FID Diversity Precision Recall user study preference
Baselines: DreamFusion adaptation ElePose MotionBERT
Builds on: Motion Diffusion Model (MDM) DreamFusion ancestral sampling SMPLify
Closest papers by meaning
- Lifting Motion to the 3D World via 2D Diffusion (CVPR 2025 · similarity 0.89)
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.87)
- 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation (ICLR 2025 · similarity 0.86)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.86)
- Single Motion Diffusion (ICLR 2024 · similarity 0.86)
- RoHM: Robust Human Motion Reconstruction via Diffusion (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.