Research map / Video / Video generation
HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HumanDreamer, a decoupled text-to-pose then pose-to-video framework for controllable human-motion video generation, trained on a new 1.2M text-pose dataset MotionVid with a MotionDiT and LAMA alignment loss.
Problem
Human-motion video generation is hard to learn directly from text, and existing pose-guided methods require poses extracted from existing videos, limiting flexibility.
Approach
Decouple generation into Text-to-Pose (a diffusion transformer, MotionDiT, generating 2D pose sequences from text, with a contrastive text-pose encoder CLoP providing a latent semantic alignment loss) and Pose-to-Video (a pose-conditioned video diffusion model built on CogVideoX).
Key results
- On the MotionVid subset, MotionDiT reaches FID 149.007 vs 396.949 for MLD, and R-precision top-1 0.451 vs 0.318.
- Ablation: adding local aggregation, global attention and LAMA loss reduces FID from 283.091 (vanilla DiT) to 149.007.
- Scaling from 50K to 1.25M samples raises R-precision top-1 from 0.451 to 0.513.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MotionDiT instead of MLD | Text-to-Pose generation · MotionVid subset | FID: 62.4% | Helped |
| use MotionDiT instead of MLD | Text-to-Pose generation · MotionVid subset | Rp-top1: 41.8% | Helped |
| use MotionDiT instead of MLD | Text-to-Pose generation · MotionVid subset | Rp-top2: 26.3% | Helped |
| use MotionDiT instead of MLD | Text-to-Pose generation · MotionVid subset | Rp-top3: 18.3% | Helped |
| add local feature aggregation (ablation) | MotionDiT · MotionVid subset | FID: 283.091 -> 183.213 | Helped |
| add global attention to local feature aggregation (ablation) | MotionDiT · MotionVid subset | FID: 183.213 -> 162.022 | Helped |
| add LAMA loss to local feature aggregation and global attention (ablation) | MotionDiT · MotionVid subset | FID: 162.022 -> 149.007 | Helped |
| increase training data from 50K to 1.25M samples | MotionDiT · MotionVid | Rp-top1: 0.451 -> 0.513 | Helped |
Limitations
- Text-to-Pose evaluated only on a 50K MotionVid subset and compared against 3D motion methods adapted to 2D poses, since Tender's code and data are unavailable.
- Dataset restricted to single-person scenes with visible faces after filtering.
- Pose-to-Video results are mainly qualitative; no quantitative video metrics reported in the main text.
Experimental setup
Datasets: MotionVid Kinetics-400 Kinetics-700 ActivityNet-200 Something-Something V2 CAER DFEW UBody Charades Charades-Ego HAA500 HMDB51
Benchmarks: MotionVid subset
Metrics: FID R-precision top1/top2/top3 Diversity MultiModality MM Distance
Baselines: T2M-GPT PriorMDM MLD Mochi1 CogVideoX
Builds on: DiT Latte CLIP MLD CogVideoX
Closest papers by meaning
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.91)
- Move-in-2D: 2D-Conditioned Human Motion Generation (CVPR 2025 · similarity 0.89)
- TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation (CVPR 2025 · similarity 0.88)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- SnapMoGen: Human Motion Generation from Expressive Texts (NeurIPS 2025 · similarity 0.87)
- OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.