Research map / Video / Video generation
Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Loopy is an end-to-end audio-only conditioned video diffusion model for portrait avatars that uses inter/intra-clip temporal layers, a temporal segment module for long motion context, and an audio-to-latents module, avoiding spatial motion constraints.
Problem
Audio-driven portrait video generation with diffusion models relies on auxiliary spatial constraints (face locators, speed layers, movement templates) to stay stable, which limits natural and diverse head and facial motion.
Approach
Extend the temporal receptive field to over 100 motion frames via a temporal segment module with separate inter-clip temporal layers, and map strongly motion-correlated head/expression variance conditions into a shared motion-latent space via an audio-to-latents module so audio alone can drive motion at test time.
Key results
- On CelebV-HQ, Loopy achieves IQA 3.780, Sync-C 4.849, FVD-R 49.153 and FID 33.204, best among compared methods.
- On RAVDESS, Loopy reaches FVD-R 16.134 and E-FID 3.132, outperforming Hallo, EchoMimic, VExpress and SadTalker.
- Ablation on HDTF/openset shows removing the inter-clip temporal layer causes the largest drop (e.g., HDTF FVD-R rises from 10.443 to 11.595 and openset IQA falls from 4.507 to 4.335).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Loopy instead of Hallo | Audio-driven portrait video generation · CelebV-HQ | IQA: 3.505 -> 3.780 | Helped |
| Use Loopy instead of Hallo | Audio-driven portrait video generation · CelebV-HQ | Sync-C: 4.130 -> 4.849 | Helped |
| Use Loopy instead of SadTalker | Audio-driven portrait video generation · RAVDESS | FVD-R: 22.516 -> 16.134 | Helped |
| Use Loopy instead of EchoMimic | Audio-driven portrait video generation · HDTF | FVD-R: 18.718 -> 10.443 | Helped |
| Use Loopy instead of EchoMimic | Audio-driven portrait video generation · Openset | IQA: 4.447 -> 4.507 | Helped |
| Add inter-clip temporal module (ablation) | Loopy · Openset | IQA: 4.335 -> 4.507 | Helped |
| Add inter-clip temporal module (ablation) | Loopy · HDTF | IQA: 3.769 -> 4.017 | Helped |
| Add temporal segment module (ablation) | Loopy · Openset | IQA: 4.386 -> 4.507 | Helped |
| Add temporal segment module (ablation) | Loopy · HDTF | IQA: 3.908 -> 4.017 | Helped |
| Add audio-to-latents module (ablation) | Loopy · Openset | IQA: 4.428 -> 4.507 | Helped |
| Add audio-to-latents module (ablation) | Loopy · HDTF | IQA: 3.918 -> 4.017 | Helped |
| Use dual temporal layers rather than one temporal layer with 20 motion frames (ablation) | Loopy · Openset | IQA: 4.072 -> 4.507 | Helped |
Surprising findings
- Hallo retrained on the authors' dataset did not improve over the original Hallo, suggesting the method design rather than data drives the gains.
- VExpress achieves the best lip-sync on RAVDESS but its outputs show little dynamic motion, so high Sync-C can reflect static videos.
Limitations
- Relies on a proprietary 174-hour training set that is not released.
- Motion frames can introduce instability when motion-frame length exceeds the current clip without proper handling, as the single-temporal-layer 20-frame variant degraded smoothness.
Experimental setup
Datasets: CelebV-HQ RAVDESS HDTF
Benchmarks: CelebV-HQ RAVDESS HDTF openset test set
Metrics: IQA Sync-C Sync-D FVD FID E-FID Smoothness Glo Exp DGlo
Baselines: SadTalker Hallo VExpress EchoMimic DiffusedHead EMO
Builds on: Stable Diffusion AnimateAnyone EMO DDIM wav2vec
Closest papers by meaning
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.89)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.88)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
- Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · similarity 0.88)
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.88)
- Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.