Research map / Video / Video generation
X-Dyna: Expressive Dynamic Human Image Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-Dyna, a zero-shot diffusion pipeline that animates a single human image with body and facial-expression motion from a driving video while synthesizing dynamics in the human and background, using a Dynamics-Adapter, a face expression ControlNet, and mixed human/scene video training.
Problem
Human image animation from a single reference image often yields rigid human motion and static backgrounds, losing fluid dynamic details (hair, garments, environmental effects) and expression fidelity.
Approach
Replace the heavy ReferenceNet appearance branch with a lightweight Dynamics-Adapter that adds a trainable cross-frame attention residual (trainable query and zero-initialized output projection) from the denoised reference latent, plus an identity-disentangled face-expression ControlNet trained on cross-identity face patches and a harmonic mix of human and natural-scene videos.
Key results
- DTFVD of 1.518 vs. 2.310 for the best baseline (Animate-Anyone) on dynamics texture generation.
- Face-Cos of 0.497 and Face-Det of 94.8%, the best among compared methods in Table 2.
- User study overall score 4.09 vs. 2.89 for the best baseline (Animate-Anyone), with 100 raters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use X-Dyna instead of the strongest listed baseline | human image animation · Pexels dynamics test set | FG-DTFVD: 1.753 -> 0.900 | Helped |
| use X-Dyna instead of the strongest listed baseline | human image animation · Pexels dynamics test set | BG-DTFVD: 1.901 -> 1.101 | Helped |
| use X-Dyna instead of the strongest listed baseline | human image animation · Pexels dynamics test set | DTFVD: 2.310 -> 1.518 | Helped |
| use X-Dyna instead of MimicMotion | human image animation · TikTok test set | cd-FVD (foreground): 150.01 -> 325.35 | Hurt |
| use X-Dyna instead of MagicPose | human image animation · TikTok test set | FID (background): 24.43 -> 25.59 | Hurt |
| replace Dynamics-Adapter with ReferenceNet (ablation) | X-Dyna · Self-collected dynamics test set | DTFVD: 1.518 -> 2.823 | Hurt |
| replace Dynamics-Adapter with IP-Adapter (ablation) | X-Dyna · Self-collected dynamics test set | DTFVD: 1.518 -> 4.851 | Hurt |
| use face landmarks with the pose skeleton instead of S-Face ControlNet (ablation) | X-Dyna · Self-collected dynamics test set | DTFVD: 1.518 -> 1.589 | Hurt |
| remove S-Face ControlNet fine-tuning (ablation) | X-Dyna · Self-collected dynamics test set | DTFVD: 1.518 -> 1.550 | Hurt |
| remove Harmonic Data Fusion Training (ablation) | X-Dyna · Self-collected dynamics test set | DTFVD: 1.518 -> 1.652 | Hurt |
Surprising findings
- Replacing the Dynamics-Adapter with a ReferenceNet or IP-Adapter in ablation lowered DTFVD sharply (2.823 and 4.851 vs 1.518), showing strong appearance constraints kill dynamics.
Limitations
- Identity and appearance may not be preserved under extreme zooming or when target pose deviates strongly from reference.
- Struggles to generate accurate hand poses.
- Built on SD 1.5 rather than stronger video backbones.
Experimental setup
Datasets: custom human motion video dataset (900h) Skyscape TikTok test split Pexels self-collected test set
Benchmarks: TikTok Pexels self-collected test set
Metrics: PSNR SSIM L1 LPIPS FID cd-FVD Face-Cos Face-Det DTFVD FG-DTFVD
Baselines: MagicAnimate Animate-Anyone MagicPose MimicMotion
Builds on: Stable Diffusion 1.5 AnimateDiff ControlNet ReferenceNet I2V-Adapter
Closest papers by meaning
- X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention (ICLR 2025 · similarity 0.89)
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.88)
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.87)
- Animate-X: Universal Character Image Animation with Enhanced Motion Representation (ICLR 2025 · similarity 0.87)
- Free-viewpoint Human Animation with Pose-correlated Reference Selection (CVPR 2025 · similarity 0.86)
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.