Research map / Image generation / Editing and style transfer
EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EchoMimic, a diffusion-based portrait animation model trained jointly on audio and facial landmark conditions so it can be driven by audio, landmarks, or both, with a timestep-aware pixel-space spatial loss.
Problem
Audio-driven talking-head portrait animation is unstable due to weak audio signals, while landmark-driven animation is stable but unnatural from over-constraint; existing methods use only one condition type.
Approach
Jointly train a latent diffusion video model conditioned on audio (Wav2Vec features via audio attention) and facial landmarks (convolutional landmark encoder added to latents), with random landmark selection (dropping facial parts) so it supports audio-only, landmark-only, and audio+selected-landmark driving; adds a timestep-weighted pixel-space L2+LPIPS loss and part-aware motion synchronization
Key results
- On HDTF, EchoMimic (audio-driven) reports FID 29.136 and FVD 492.784, best among compared methods, vs Hallo's 37.659 and 501.074.
- On CelebV-HQ, EchoMimic achieves the lowest FID (63.258) and E-FID (2.723), while Hallo has the best FVD and SSIM.
- Landmark-only driving on HDTF gives FVD 156.537 and SSIM 0.889, versus audio-only FVD 492.784 and SSIM 0.812, showing landmarks give the most stable outputs but less freedom.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use landmark-only instead of audio-only driving (ablation) | EchoMimic · HDTF | FID: 29.136 -> 22.970 | Helped |
| Use landmark-only instead of audio-only driving (ablation) | EchoMimic · HDTF | FVD: 492.784 -> 156.537 | Helped |
| Use landmark-only instead of audio-only driving (ablation) | EchoMimic · HDTF | SSIM: 0.812 -> 0.889 | Helped |
| Use landmark-only instead of audio-only driving (ablation) | EchoMimic · HDTF | E-FID: 1.112 -> 1.057 | Helped |
| Use audio plus selected landmarks instead of audio-only driving (ablation) | EchoMimic · HDTF | FID: 29.136 -> 22.981 | Helped |
| Use audio plus selected landmarks instead of audio-only driving (ablation) | EchoMimic · HDTF | FVD: 492.784 -> 181.741 | Helped |
| Use audio plus selected landmarks instead of audio-only driving (ablation) | EchoMimic · HDTF | SSIM: 0.812 -> 0.885 | Helped |
| Use audio plus selected landmarks instead of audio-only driving (ablation) | EchoMimic · HDTF | E-FID: 1.112 -> 1.093 | Helped |
| Use landmark-only instead of audio plus selected landmarks (ablation) | EchoMimic · HDTF | FID: 22.981 -> 22.970 | Mixed |
Surprising findings
- Landmark-only driving yields closer resemblance to the original videos than audio-only driving, which the authors attribute to strong landmark constraints.
- On the self-collected dataset EchoMimic does not have the best SSIM (0.691 vs SadTalker's 0.699).
Limitations
- Architecture is an extension of image-based Stable Diffusion rather than a native video model.
- Inference is not real-time; acceleration methods are suggested as future work.
- Training data is largely self-collected and not released, limiting reproducibility.
Experimental setup
Datasets: HDTF CelebV-HQ self-collected talking head videos (~540 hours)
Benchmarks: HDTF CelebV-HQ self-collected dataset
Metrics: FID FVD SSIM E-FID
Baselines: SadTalker AniPortrait V-Express Hallo
Builds on: Stable Diffusion AnimateDiff Animate Anyone Wav2Vec MediaPipe
Closest papers by meaning
- EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation (CVPR 2025 · similarity 0.89)
- EchoShot: Multi-Shot Portrait Video Generation (NeurIPS 2025 · similarity 0.88)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.87)
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.87)
- FaceShot: Bring Any Character into Life (ICLR 2025 · similarity 0.86)
- Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.