Research map / Video / Video generation
Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the pretrained CogVideoX video DiT to audio-driven portrait animation with an identity reference network, audio cross-attention, and motion-frame extrapolation for long videos.
Problem
Portrait image animation struggles with non-frontal poses, dynamic foreground/background objects, and immersive realistic backgrounds, and prior U-Net-based identity, audio, and long-video methods do not transfer to DiT video backbones.
Approach
Applies a pretrained DiT video generator (CogVideoX) to portrait animation, using a identity reference network that reuses the causal 3D VAE and 42 transformer layers to inject identity features into self-attention, cross-attention for wav2vec audio conditioning, and last-frame motion frames for long-video extrapolation.
Key results
- On HDTF, achieves best FID (20.359) and FVD (160.838), but lower Sync-C (7.252) than Hallo (7.750) and SadTalker (7.885).
- On Celeb-V, FID 43.271 and FVD 355.272 are best among compared methods, while Sync-C is 6.527 vs 7.191 for Hallo.
- On the wild dataset, highest subject dynamic (13.286) and background dynamic (4.481) and best Sync-C (6.154).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use cross-attention instead of AdaLN for audio conditioning (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | Sync-C: 1.374 -> 4.601 | Helped |
| use cross-attention instead of AdaLN-zero for audio conditioning (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | Sync-C: 1.398 -> 4.601 | Helped |
| use cross-attention instead of self-attention for audio conditioning (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | Sync-C: 1.345 -> 4.601 | Helped |
| combine face attention with the identity reference network (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | FID: 28.789 -> 23.458 | Helped |
| add an identity reference network to face attention (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | FID: 57.541 -> 23.458 | Helped |
| add an identity reference network to face adaptive norm (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | FID: 150.720 -> 23.458 | Helped |
| add face attention and identity reference network (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | Subject consistency: 0.977 -> 0.988 | Helped |
| use 1 motion frame instead of 2 (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | Sync-C: 4.601 -> 6.889 | Helped |
| use 1 motion frame instead of 2 (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | FID: 23.458 -> 24.040 | Hurt |
| use 4 motion frames instead of 2 (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | FID: 23.458 -> 24.459 | Hurt |
| use 8 motion frames instead of 2 (ablation) | Hallo3 · HDTF/Celeb-V ablation evaluation | FID: 23.458 -> 27.303 | Hurt |
| use Hallo3 rather than Hallo | portrait animation · HDTF | FID: 20.545 -> 20.359 | Helped |
Surprising findings
- Cross-attention audio conditioning gives much better lip sync than AdaLN or self-attention, which all score Sync-C near 1.4 in ablation.
- Using only 1 motion frame gives the best Sync-C/Sync-D, while 2 frames were chosen for the main setting despite worse lip sync.
Limitations
- Lip synchronization is lower than several baselines on HDTF and Celeb-V.
- Evaluation on wild dataset is small (34 cases) and the dataset is not released.
- Training data includes large amounts of noisy YouTube and movie footage requiring heavy curation.
Experimental setup
Datasets: HDTF YouTube videos movie videos Celeb-V
Benchmarks: HDTF Celeb-V Wild dataset (34 cases)
Metrics: FID FVD Sync-C Sync-D E-FID subject dynamic background dynamic subject consistency
Baselines: SadTalker DreamTalk AniPortrait Hallo
Builds on: CogVideoX wav2vec 2.0 T5 InsightFace 3D RoPE
Closest papers by meaning
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.91)
- Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation (ICLR 2025 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.88)
- Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR 2024 · similarity 0.88)
- Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.