Research map / Video / Video generation
Move-in-2D: 2D-Conditioned Human Motion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Move-in-2D generates 3D human motion (SMPL sequences with camera translation) conditioned on a single scene image and a text prompt using a multi-conditional diffusion transformer trained on a 300k-video dataset (HiC-Motion), then uses the motion to drive video generation.
Problem
Generating human motion sequences that are compatible with a 2D scene image (and text prompt) without 3D scene reconstruction, for use as control signals in human video generation.
Approach
A diffusion transformer that conditions on text (CLIP) and scene image (DINO) via in-context tokens, with AdaLN for timestep, predicting SMPL pose plus camera translation so motion projects onto the 2D scene; trained on a large internet-video-derived motion dataset with a two-stage schedule that disentangles camera motion.
Key results
- Ours achieves FID 44.639 and accuracy 0.661, versus MDM+ at FID 46.035 and accuracy 0.620 and MDM at FID 164.595.
- In GPT-4o automated scoring, Ours reaches total 9.10 (scene-align 3.55) versus MLD at 6.70 and HUMANISE at 4.95.
- Ours-scene gets higher multimodality (21.320) than Ours (20.130), while Ours has higher accuracy than Ours-scene by about 37%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train MDM on HiC-Motion rather than HumanML3D | MDM · HiC-Motion test set | FID: 164.595 -> 46.035 | Helped |
| Train MDM on HiC-Motion rather than HumanML3D | MDM · HiC-Motion test set | Accuracy: 0.325 -> 0.620 | Helped |
| Add text conditioning to scene-conditioned generation | Ours-scene · HiC-Motion test set | Accuracy: 0.482 -> 0.661 | Helped |
| Add text conditioning to scene-conditioned generation | Ours-scene · HiC-Motion test set | Multimodality: 21.320 -> 20.130 | Hurt |
| Replace In-Context scene conditioning with Cross-Attn (ablation) | Move-in-2D · HiC-Motion test set | FID: 44.639 -> 47.656 | Hurt |
| Replace In-Context scene conditioning with Cross-Attn (ablation) | Move-in-2D · HiC-Motion test set | Accuracy: 0.661 -> 0.567 | Hurt |
| Replace AdaLN timestep conditioning with In-Context (ablation) | Move-in-2D · HiC-Motion test set | FID: 44.639 -> 62.927 | Hurt |
| Replace AdaLN timestep conditioning with In-Context (ablation) | Move-in-2D · HiC-Motion test set | Accuracy: 0.661 -> 0.554 | Hurt |
| Replace AdaLN timestep conditioning with In-Context and In-Context scene conditioning with Cross-Attn (ablation) | Move-in-2D · HiC-Motion test set | FID: 44.639 -> 66.827 | Hurt |
| Replace AdaLN timestep conditioning with In-Context and In-Context scene conditioning with Cross-Attn (ablation) | Move-in-2D · HiC-Motion test set | Accuracy: 0.661 -> 0.519 | Hurt |
Surprising findings
- Scene-conditioned baselines HUMANISE and SceneDiff underperform text-only MLD and MDM, attributed to training on limited synthetic point clouds.
Limitations
- Does not control camera movement in generated motions.
- Two-pass video generation pipeline is not jointly optimized.
- Training data comes from a proprietary internal 30M-video pool, so the dataset is not public.
Experimental setup
Datasets: HiC-Motion (300k internet videos with 4D-Humans SMPL annotations) HumanML3D HUMANISE
Benchmarks: HiC-Motion held-out test set (957 samples)
Metrics: FID accuracy diversity multimodality VLM score (scene-align, text-align, quality)
Baselines: MDM MLD SceneDiff HUMANISE MDM+ SVD
Builds on: MDM DDPM Diffusion Transformer (DiT) 4D-Humans Champ
Closest papers by meaning
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.90)
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.89)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.89)
- Make Pixels Dance: High-Dynamic Video Generation (CVPR 2024 · similarity 0.89)
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.89)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.