Research map / Video / Video generation
DisCo: Disentangled Control for Realistic Human Dance Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DISCO builds a disentangled-control diffusion model (VAE background branch, CLIP-local foreground cross-attention, pose ControlNet) for referring human dance image/video generation, pre-trained on human images for better generalization to unseen humans.
Problem
Human dance generation for real-world social media content requires generalizing to unseen humans, backgrounds, and poses, and allowing arbitrary composition of subject, background, and pose from different sources; existing motion-transfer and ControlNet-based methods fail at this.
Approach
Disentangle the three conditions: foreground human via CLIP local image embeddings in cross-attention, background via a VAE-encoder ControlNet branch, and pose via a separate ControlNet branch; then pre-train on large-scale human images by reconstructing whole images from foreground and background features (human attribute pre-training) before pose fine-tuning.
Key results
- DISCO without HAP achieves FID 61.06 vs DreamPose 79.46; with HAP and CFG FID drops to 30.75 and scaled-up DISCO reaches FID 28.31 vs best baseline TPS 53.78.
- Scaled-up DISCO reaches FID-VID 55.17 vs MRAA 66.36; adding temporal modeling lowers FID-VID to 29.37.
- User study with 50 participants: over 80% of users rated DISCO images/videos 4 or above on a 0-5 scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add human attribute pre-training (ablation) | DISCO · TikTok test | FID: 61.06 -> 38.19 | Helped |
| Add classifier-free guidance | DISCO with HAP · TikTok test | FID: 38.19 -> 30.75 | Helped |
| Scale fine-tuning to ∼600 TikTok-style videos | DISCO with HAP and CFG · TikTok test | FID-VID: 59.90 -> 55.17 | Helped |
| Add temporal modeling (ablation) | DISCO with HAP and CFG · TikTok test | FID-VID: 59.90 -> 34.19 | Helped |
| Use one ControlNet for foreground and background instead of disentangled controls (ablation) | DISCO · TikTok test | FID: 61.06 -> 65.14 | Hurt |
| Use attention for foreground and background instead of disentangled controls (ablation) | DISCO · TikTok test | FID: 61.06 -> 80.50 | Hurt |
| Replace local CLIP foreground features with global CLIP features (ablation) | DISCO · TikTok test | FID: 61.06 -> 63.92 | Hurt |
| Add VAE features to local CLIP foreground features (ablation) | DISCO · TikTok test | FID: 61.06 -> 59.74 | Helped |
| Pre-train on TikTok human images (ablation) | DISCO · TikTok test | FID: 61.06 -> 50.68 | Helped |
| Increase pre-training data by adding COCO (ablation) | DISCO pretrained on TikTok · TikTok test | FID: 50.68 -> 48.89 | Helped |
| Increase pre-training data by adding SHHQ (ablation) | DISCO pretrained on TikTok and COCO · TikTok test | FID: 48.89 -> 44.13 | Helped |
| Increase pre-training data by adding DeepFashion2 and LAION (ablation) | DISCO pretrained on TikTok, COCO, and SHHQ · TikTok test | FID: 44.13 -> 38.19 | Helped |
Surprising findings
- ControlNet-only architecture achieves better FVD than the disentangled design but worse visual consistency of human attributes and background.
- Simply combining ControlNet and cross-attention for the whole reference image performs worse than the disentangled design.
Limitations
- Cannot handle hand posture well without fine-grained hand pose control.
- Hard to apply to multi-person scenarios.
- Not designed for human-object interactions.
Experimental setup
Datasets: TikTok COCO SHHQ DeepFashion2 LAION Taichi NTU-120 AIST++ Everybody Dance Now
Benchmarks: TikTok Taichi NTU-120
Metrics: FID SSIM PSNR LPIPS L1 FID-VID FVD user study score
Baselines: FOMM MRAA TPS DreamPose T2I Adaptor
Builds on: Stable Diffusion ControlNet DreamPose SD Image Variations
Closest papers by meaning
- Bidirectional Autoregessive Diffusion Model for Dance Generation (CVPR 2024 · similarity 0.87)
- DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning (AAAI 2024 · similarity 0.86)
- Breaking the Group Size Barrier: Parameter-Efficient Group Dance Generation with Chain-of-Dancers (NeurIPS 2026 · similarity 0.86)
- DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance (CVPR 2024 · similarity 0.86)
- TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation (CVPR 2025 · similarity 0.86)
- Controllable 3D Dance Generation Using Diffusion-Based Transformer U-Net (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.