Research map / Image generation / Editing and style transfer
MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MagicPose retargets 2D human poses and facial expressions from a reference image using a Stable Diffusion plug-in with a trainable appearance-control UNet (multi-source self-attention) and a pose ControlNet trained in two stages.
Problem
Human pose and facial expression retargeting that preserves the identity, skin tone, clothing and background of a reference image while generalizing to unseen identities, styles and poses without per-target fine-tuning.
Approach
Replicates the Stable Diffusion UNet as a trainable appearance-control branch whose keys and values are concatenated with the frozen UNet's self-attention (multi-source self-attention), pretrained first for appearance, then jointly fine-tuned with a pose ControlNet so pose control is disentangled from appearance.
Key results
- On TikTok, MagicPose reaches FID 25.50 and Face-Cos 0.426, versus 30.75 FID and 0.166 Face-Cos for DisCo (dagger variant).
- In a 100-participant user study, 73% of votes chose MagicPose for preserving identity and appearance, versus 16% for DisCo.
- Appearance control pretraining raises Face-Cos from 0.038 to 0.397 in the ablation, showing its large effect on identity preservation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MagicPose instead of DisCo† | Human pose and expression retargeting · TikTok test | Face-Cos: +0.260 | Helped |
| Use MagicPose instead of DisCo† | Human pose and expression retargeting · TikTok test | FID: 30.75 -> 25.50 | Helped |
| Use MagicPose instead of DisCo† | Human pose and expression retargeting · TikTok test | FID-VID: 59.90 -> 46.30 | Helped |
| Add Appearance Control Pretraining (ablation) | MagicPose · TikTok | Face-Cos: +944.73% | Helped |
| Add Appearance Control Pretraining (ablation) | MagicPose · TikTok | SSIM: +149.82% | Helped |
| Add Appearance-disentangled Pose Control (ablation) | MagicPose with Appearance Control Pretraining · TikTok | Face-Cos: +7.30% | Helped |
| Add Appearance-disentangled Pose Control (ablation) | MagicPose with Appearance Control Pretraining · TikTok | SSIM: +3.43% | Helped |
| Add Image-CFG (ablation) | MagicPose · TikTok | Face-Cos: +56.62% | Helped |
| Add Image-CFG (ablation) | MagicPose · TikTok | SSIM: +14.11% | Helped |
| Add random masking of facial landmarks and hand poses during training (ablation) | MagicPose · TikTok | Face-Cos: +2.20% | Helped |
| Add random masking of facial landmarks and hand poses during training (ablation) | MagicPose · TikTok | SSIM: +0.13% | Helped |
| Fine-tune on Everybody Dance Now train set after TikTok training | MagicPose · Everybody Dance Now test | FID: 28.64 -> 26.24; 22.59 -> 22.50; 22.21 -> 22.61; 35.43 -> 27.38; 31.72 -> 36.73; 31.24 -> 21.99 | Mixed |
Surprising findings
- Vanilla ControlNet fails to keep appearance when generating different poses, and naively combining OpenPose ControlNet with the pretrained appearance module struggles with appearance-independent pose control.
Limitations
- Relies on OpenPose, which often gives incomplete skeletons and hand/face landmarks, causing artifacts.
- Evaluation largely on TikTok, which mostly shows young women upper-body images; generalization to other demographics is limited in evaluation.
- Temporal consistency for video requires an optional AnimateDiff-based motion module that the authors found insufficient out of the box.
Experimental setup
Datasets: TikTok Everybody Dance Now self-collected out-of-domain images
Benchmarks: TikTok Everybody Dance Now
Metrics: FID SSIM PSNR LPIPS L1 Face-Cos FID-VID user preference
Baselines: FOMM MRAA TPS DreamPose DisCo
Builds on: Stable Diffusion ControlNet DreamPose DisCo AnimateDiff
Closest papers by meaning
- MagiCapture: High-Resolution Multi-Concept Portrait Customization (AAAI 2024 · similarity 0.87)
- Cross-view Masked Diffusion Transformers for Person Image Synthesis (ICML 2024 · similarity 0.86)
- Multi-focal Conditioned Latent Diffusion for Person Image Synthesis (CVPR 2025 · similarity 0.86)
- IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation (NeurIPS 2024 · similarity 0.86)
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.86)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.