Research map / Video / Video generation
Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the Hallo audio-driven portrait animation model to hour-long, 4K, text-prompt-controllable videos via patch-drop and Gaussian noise augmentation of motion frames, plus a VQ-based temporal-alignment super-resolution module.
Problem
Generating long-duration (minutes to hours) audio-driven portrait animations that keep identity and temporal coherence without appearance drift, at high (4K) resolution, with controllable expressions.
Approach
Corrupt the appearance content of conditioning motion frames with patch dropping plus Gaussian noise so the model takes appearance from the reference image and only motion from preceding frames, and add a temporal-alignment layer to a codebook-prediction (CodeFormer-initialized) super-resolution module for 4K output; also inject CLIP text embeddings via adaptive layer norm for expression control.
Key results
- On the CelebV set (512x512, 4-minute videos), Hallo2 reaches FID 37.944 and FVD 477.412 versus 82.715 and 1088.158 for the Hallo baseline, with patch drop plus Gaussian noise giving the best scores.
- On the Wild dataset, Hallo2 achieves FID 24.072 and Sync-C 6.760, the lowest FID among compared methods, with the best E-FID of 33.316.
- Patch drop with patch size 1 and drop rate 0.25 gives the best FID 38.518 and FVD 491.338 on CelebV, compared with 82.715 and 1088.158 without it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Hallo2 method instead of the strongest listed baseline | Portrait animation · CelebV | FID: 41.852 -> 37.944 | Helped |
| Use the Hallo2 method instead of the strongest listed baseline | Portrait animation · CelebV | Sync-C: 7.069 -> 6.928 | Hurt |
| Use the Hallo2 method instead of the strongest listed baseline | Portrait animation · Wild | FID: 24.301 -> 24.072 | Helped |
| Use the Hallo2 method instead of the strongest listed baseline | Portrait animation · Wild | FVD: 344.000 -> 360.192 | Hurt |
| Add patch-drop augmentation (patch size 1) rather than no patch drop (ablation) | Hallo2 long-duration animation · CelebV | FID: 82.715 -> 38.518 | Helped |
| Vary patch size from 1 to 4 or 16 (ablation) | Hallo2 long-duration animation · CelebV | FID: 38.518 -> 39.615; 38.518 -> 44.172 | Hurt |
| Set patch-drop rate to 0.25 rather than 0.1 or 0.5 (ablation) | Hallo2 long-duration animation · CelebV | FID: 41.687 -> 38.518; 39.642 -> 38.518 | Helped |
| Add Gaussian noise to patch-drop augmentation (ablation) | Hallo2 long-duration animation · CelebV | FID: 38.518 -> 37.944 | Helped |
| Add Gaussian noise to patch-drop augmentation (ablation) | Hallo2 long-duration animation · CelebV | Sync-C: 6.766 -> 6.928 | Helped |
| Use Gaussian noise alone instead of no augmentation (ablation) | Hallo2 long-duration animation · CelebV | FID: 82.715 -> 78.283 | Helped |
| Add Gaussian noise without patch drop (ablation) | Hallo2 long-duration animation · CelebV | FVD: 1088.158 -> 984.876 | Helped |
Surprising findings
- Gaussian noise alone barely helps (FID 78.283 vs 82.715 baseline), while patch drop alone yields most of the gain, and the combination is best.
- On the Wild dataset, Hallo2 reports the highest FVD (360.192) among the compared methods, which the authors frame as superior despite it being the worst-looking in the FVD column relative to AniPortrait and SadTalker.
Limitations
- Reliance on a single reference image limits expression and pose diversity.
- Patch-drop augmentation may introduce artifacts.
- 4K generation is computationally demanding and not real-time.
Experimental setup
Datasets: HDTF CelebV Wild VFHQ
Benchmarks: HDTF CelebV Wild
Metrics: FID FVD Sync-C Sync-D E-FID
Baselines: Audio2Head SadTalker EchoMimic AniPortrait Hallo
Builds on: Hallo AnimateDiff CodeFormer VQGAN wav2vec
Closest papers by meaning
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.90)
- Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency (ICLR 2025 · similarity 0.88)
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.87)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.86)
- EverAnimate: Minute-Scale Human Animation via Latent Flow Restoration (NeurIPS 2026 · similarity 0.86)
- Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.