Research map / Image generation / Autoregressive and latent generation
DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio temporal filtering (ablation) | DiffTalk · Test Set A and Test Set B | SyncNet confidence: 5.484 -> 6.287; 5.197 -> 5.387 | Helped |
| Add audio temporal filtering (ablation) | DiffTalk · Test Set A and Test Set B | PSNR: 33.67 -> 34.17; 32.70 -> 32.73 | Mixed |
| Use the previous generated frame instead of a random reference image (ablation) | DiffTalk inference · Test Set A | LPIPS: 0.024 -> 0.023 | Helped |
| Use the previous generated frame instead of a random reference image (ablation) | DiffTalk inference · Test Set B | LPIPS: 0.031 -> 0.030 | Helped |
| Use the previous generated frame instead of a random reference image (ablation) | DiffTalk inference · Test Set A and Test Set B | SyncNet confidence: 6.287 -> 6.662; 5.387 -> 5.999 | Helped |
| Add landmarks to audio conditioning (ablation) | DiffTalk · Novel-identity test examples | face alignment | Helped |
| Combine a random reference image with a masked ground-truth image (ablation) | DiffTalk · Novel-identity test examples | identity and mouth-appearance fidelity | Helped |
| Use audio alone instead of reference and landmark conditions (ablation) | DiffTalk · Novel-identity test examples | identity generalization | Hurt |
| Use a random reference image without a masked ground-truth image (ablation) | DiffTalk · Novel-identity test examples | training convergence | Hurt |
Closest papers by meaning
- Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency (ICLR 2025 · similarity 0.86)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.86)
- MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer (AAAI 2025 · similarity 0.85)
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.85)
- EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions (AAAI 2025 · similarity 0.85)
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.