Research map / Video / Video generation
Diffusion-based human image animation
Animates portraits and human images with video diffusion, controllable motion and camera conditions.
126 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Video generation, keyword cluster “audio · editing · talking”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 103 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 7.0% to 9.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Video2GIF: Automatic Generation of Animated GIFs From Video (CVPR 2016 · 139 citations)
- 2018 · PVL: A Framework for Navigating the Precision-Variety Trade-Off in Automated Animation of Smiles (AAAI 2018)
- 2019 · Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation (AAAI 2019 · 34 citations)
- 2020 · G3AN: Disentangling Appearance and Motion for Video Generation (CVPR 2020 · 83 citations)
- 2021 · High-Fidelity Neural Human Motion Transfer From Monocular Video (CVPR 2021 · 33 citations)
- 2022 · NeMF: Neural Motion Fields for Kinematic Animation (NeurIPS 2022 · 12 citations)
- 2023 · VideoComposer: Compositional Video Synthesis with Motion Controllability (NeurIPS 2023 · 43 citations)
- 2024 · MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (CVPR 2024 · 143 citations)
- 2025 · Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · 29 citations)
- 2026 · Animating the Uncaptured: Humanoid Mesh Animation with Video Diffusion Models (ICLR 2026 · 1 citation)
Earliest papers here
- Video2GIF: Automatic Generation of Animated GIFs From Video (CVPR 2016 · 139 citations)
- PVL: A Framework for Navigating the Precision-Variety Trade-Off in Automated Animation of Smiles (AAAI 2018)
- Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation (AAAI 2019 · 34 citations)
What has been tried
71 papers here report 715 measured changes (a component added, removed or swapped, compared with a baseline): helped 78%, no clear effect 3%, hurt 18%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| remove temporal attention layers | TikTok · FVD | 179.07 -> 247.30 | Hurt | MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (CVPR 2024 · 143 citations) |
| use ReferenceNet instead of CLIP-only conditioning | UBC fashion video dataset · SSIM | 0.897 -> 0.931 | Helped | Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · 102 citations) |
| use cross-attention instead of AdaLN for audio conditioning | HDTF/Celeb-V ablation evaluation · Sync-C | 1.374 -> 4.601 | Helped | Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · 29 citations) |
| add a neural network rather than directly optimize control-point coordinates | Ablation evaluation · sketch-to-video consistency | 0.926 ± 0.007 -> 0.965 ± 0.003 | Helped | Breathing Life Into Sketches Using Text-to-Video Priors (CVPR 2024 · 27 citations) |
| Add batch-overlapped temporal denoising | Test clips · FVD | 344.85 -> 139.82 | Helped | Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework (CVPR 2024 · 25 citations) |
| add masked temporal LoRA fine-tuning | VBench · Temporal Smoothness | 74.27% -> 97.32% | Helped | ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning (CVPR 2025 · 17 citations) |
| add control signals | NuScenes · ADE | 3.24 -> 3.07 | Helped | GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control (CVPR 2025 · 16 citations) |
| Add global projection after CLIP rather than using the CLIP image embedding directly | Stylized video generation · UMT-Score | 0.892 -> 2.337 | Helped | StyleMaster: Stylize Your Video with Artistic Generation and Translation (CVPR 2025 · 16 citations) |
| add WebVid-Motion data and the motion-augmented module | Regional image animation benchmark · Text-Align | 0.3659 -> 0.7981 | Helped | Follow-Your-Click: Open-domain Regional Image Animation via Motion Prompts (AAAI 2025 · 15 citations) |
| Use the temporal consistency network | INSTA video dataset · warping error | 0.0006 -> 0.0003 | Helped | Real-time 3D-aware Portrait Video Relighting (CVPR 2024 · 14 citations) |
| add motion memory bank | Cross Reenactment · FVD | 361.94 -> 333.48 | Helped | HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation (CVPR 2025 · 13 citations) |
| Use distinct camera and object LoRA weights instead of a shared motion control | Camera and object motion control · Motion separation | Helped | Image Conductor: Precision Control for Interactive Video Synthesis (AAAI 2025 · 11 citations) |
What reviewers will expect
Datasets and baselines named in the 50 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HDTF · 10 of 50 papers
- DAVIS · 8 of 50 papers
- CelebV-HQ · 6 of 50 papers
- VFHQ · 6 of 50 papers
- WebVid-10M · 4 of 50 papers
- UCF-101 · 4 of 50 papers
- MSRVTT · 2 of 50 papers
- RealEstate10K · 2 of 50 papers
Baselines
- DynamiCrafter · 7 of 50 papers
- AniPortrait · 7 of 50 papers
- DragAnything · 6 of 50 papers
- I2VGen-XL · 5 of 50 papers
- LivePortrait · 5 of 50 papers
- FollowYourEmoji · 5 of 50 papers
- SVD · 4 of 50 papers
- X-Portrait · 4 of 50 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 119 papers; from OpenAlex and the papers' affiliations).
- Tencent · 7 papers
- Hong Kong University of Science and Technology · 7 papers
- Tsinghua University · 6 papers
- Zhejiang University · 5 papers
- ByteDance · 5 papers
- Adobe · 5 papers
- The Chinese University of Hong Kong · 4 papers
- National University of Singapore · 4 papers
- Chinese Academy of Sciences · 4 papers
- NVIDIA · 4 papers
Use it
Download BibTeX of 28 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Diffusion-based video editing (92 papers)
- Talking face generation (81 papers)
- Video-to-audio synthesis (76 papers)
- Text-to-motion diffusion (140 papers)
Elsewhere in the topic
- Efficient video diffusion models (283 papers)
- Trajectory-guided motion control (184 papers)
- Compositional text-to-video diffusion (159 papers)
- Autoregressive long video generation (156 papers)
- World-model video generation (110 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).