Research map / Video / Video generation
DisPose: Disentangling Pose Guidance for Controllable Human Image Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DisPose, a plug-and-play hybrid ControlNet that disentangles skeleton pose guidance into a reference-based dense motion field and DIFT keypoint correspondences, improving human image animation without extra dense inputs.
Problem
Human image animation from a reference image and driving video: sparse skeleton pose gives limited control, while dense conditions (DensePose, depth, SMPL renders) constrain body shape and fail when the reference body differs from the driving video.
Approach
Derive a dense motion field from sparse keypoint motion and the reference image via condition motion propagation (CMP), and transfer DIFT diffusion features of reference keypoints along trajectories as identity correspondences, injected through a frozen-base hybrid ControlNet.
Key results
- On TikTok, MimicMotion+Ours reaches FVD 466.93 vs 598.41 for MimicMotion, and FID-FVD 10.24 vs 10.50.
- MusePose+Ours improves FVD from 866.36 to 764.00 over the MusePose baseline on TikTok.
- Image-level SSIM for MimicMotion rises from 0.749 to 0.781 with DisPose, and LPIPS drops from 0.272 to 0.242.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DisPose guidance to MimicMotion | MimicMotion · TikTok | FVD: 598.41 -> 466.93 | Helped |
| add DisPose guidance to MimicMotion | MimicMotion · Unseen dataset | Subject Consistency: 94.12 -> 94.35 | Helped |
| add motion field guidance (ablation) | Full Model · TikTok | FVD: 478.91 -> 466.93 | Helped |
| add keypoint correspondence (ablation) | Full Model · TikTok | FVD: 498.74 -> 466.93 | Helped |
| use the hybrid controller instead of inserting motion field into the denoising network (ablation) | DisPose · TikTok | FID-FVD: 10.43 -> 10.24 | Helped |
| retain the hybrid ControlNet instead of removing it (ablation) | DisPose · TikTok | FVD: 551.32 -> 466.93 | Helped |
| add CMP (ablation) | Full Model · TikTok | Subject Consistency: 93.94 -> 94.35 | Helped |
| add CMP (ablation) | Full Model · TikTok | Background Consistency: 97.83 -> 98.75 | Helped |
| add DisPose guidance to MusePose | MusePose · image-level metrics | SSIM: 0.788 -> 0.811 | Helped |
| add DisPose guidance to MusePose | MusePose · image-level metrics | LPIPS: 0.263 -> 0.238 | Helped |
Surprising findings
- Dense reference-based motion fields are estimated from the reference image rather than the driving video, so the method avoids requiring dense driving-video signals at inference.
- Trainable parameters for MimicMotion+Ours (653 MB) are about half those of the MimicMotion baseline (1454 MB), though inference time rises from 1.61 to 2.36 sec/frame.
Limitations
- Limited ability to synthesize unseen parts of characters from a single-view reference image.
- Background noise from motion field estimation errors when the reference has complex backgrounds.
- Training uses a relatively small internet video set of 3k videos and a gap exists between training optical flow and inference-time reference-based motion fields.
Experimental setup
Datasets: TikTok 3k internet human videos
Benchmarks: TikTok VBench
Metrics: FID-FVD FVD CD-FVD VBench temporal flickering VBench subject consistency VBench background consistency VBench motion smoothness VBench dynamic degree VBench imaging quality VBench aesthetic quality
Baselines: MagicPose Moore-AnimateAnyone MusePose ControlNeXt MimicMotion Champ
Builds on: ControlNet DIFT condition motion propagation MimicMotion MusePose
Closest papers by meaning
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.88)
- Free-viewpoint Human Animation with Pose-correlated Reference Selection (CVPR 2025 · similarity 0.87)
- DiffPerformer: Iterative Learning of Consistent Latent Guidance for Diffusion-based Human Video Generation (CVPR 2024 · similarity 0.87)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.87)
- Boximator: Generating Rich and Controllable Motions for Video Synthesis (ICML 2024 · similarity 0.86)
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.