Research map / 3D vision / 3D generation
Motion Diversification Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Motion Diversification Networks, which use a transformer (z-transformer) over sampled latent random variables guided by k-means motion primitives to generate diverse, accurate 3D human motion, evaluated on Human3.6M, 3DPW, HPS, and a CARLA-based DenseCity benchmark.
Problem
Generating 3D human motion that is both realistic and diverse, especially in context-rich urban scenes with multi-modal futures; existing generative models suffer mode collapse and diversity-accuracy trade-offs.
Approach
Replace DLow's affine latent mapping with a transformer that uses sampled random variables as queries and attends to encoded context plus k-means pose primitives as keys/values, producing a diverse set of latent codes decoded by a CVAE generator.
Key results
- On Human3.6M, MDN reaches APD 17.450 vs. 15.884 for STARS (prior best) with ADE 0.355 and FDE 0.442.
- On DenseCity, MDN achieves APD 16.799 vs. 11.980 for DLow, with ADE 0.584 and FDE 0.879 (3D pose).
- In a perceptual user study with 13 participants, MDN was preferred over the CARLA built-in controller 72.5% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace affine latent diversification with the z-transformer | 3D motion forecasting · Human3.6M | APD: 15.884 -> 17.450 | Helped |
| replace affine latent diversification with the z-transformer | 3D motion forecasting · 3DPW | APD: 4.111 -> 6.355 | Helped |
| replace affine latent diversification with the z-transformer | 3D motion forecasting · HPS | APD: 4.835 -> 6.943 | Helped |
| replace affine latent diversification with the z-transformer | 3D pose forecasting · DenseCity | APD: 11.980 -> 16.799 | Helped |
| add YouTube motion data during training | MDN · 3DPW | APD: 6.355 -> 8.266 | Helped |
| add YouTube motion data during training | MDN · HPS | APD: 6.943 -> 6.925 | No clear effect |
| add YouTube motion data during training | MDN · DenseCity 3D Pose | ADE: 0.584 -> 0.578 | Helped |
| add YouTube motion data during training | MDN · DenseCity 3D Pose | FDE: 0.879 -> 0.921 | Hurt |
| add YouTube motion data during training | MDN · DenseCity 2D Path | APD: 1.065 -> 1.331 | Helped |
| use joint 2D path and 3D pose prediction instead of sequential prediction | MDN · DenseCity | ADE | Helped |
| use MDN instead of the CARLA built-in pedestrian controller | closed-loop CARLA user study · CARLA social and object interaction scenarios | preference: 72.5% | Helped |
Surprising findings
- Adding YouTube data improved diversity and realism on 3DPW/HPS but hurt 3D pose accuracy on DenseCity (ADE rose in the simulated benchmark relative to the sim-only setting for 2D path only in some metrics).
Limitations
- Human3.6M is saturated and only contains isolated indoor scenes, limiting evaluation of diversity.
- Simulation (CARLA) pedestrian controller may lack realism; YouTube data hurt 3D pose accuracy on DenseCity.
- Ethical concerns about using publicly available internet videos with sensitive personal information.
Experimental setup
Datasets: Human3.6M 3DPW HPS DenseCity YouTube dataset
Benchmarks: Human3.6M 3DPW HPS DenseCity
Metrics: APD ADE FDE user preference realism score
Baselines: CVAE DLow PoseGPT HuMoR Cao et al. BeLFusion STARS MOJO GSPS
Builds on: CVAE DLow transformer Cao et al. scene-aware motion prediction
Closest papers by meaning
- Stochastic Multi-Person 3D Motion Forecasting (ICLR 2023 · similarity 0.91)
- Scene-Aware Generative Network for Human Motion Synthesis (CVPR 2021 · similarity 0.89)
- Harmonizing Stochasticity and Determinism: Scene-responsive Diverse Human Motion Prediction (NeurIPS 2024 · similarity 0.88)
- SimpliHuMoN: Simplifying Human Motion Prediction (NeurIPS 2026 · similarity 0.87)
- ProMotion: Prototypes As Motion Learners (CVPR 2024 · similarity 0.86)
- FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.