Research map / Video / Video generation
AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnyTalk, a unified one-shot talking head framework using canonical 3D keypoints and an expression consistency loss to drive cross-domain portraits (cartoon, animal, human) without paired data, and releases the AniTalk dataset of 1,250 videos.
Problem
Cross-domain talking head generation (e.g., animating a cartoon or animal face with a real human video) typically needs domain-specific networks, paired videos, and extra motion alignment modules.
Approach
A single shared model learns motion via canonical 3D keypoints from an unsupervised keypoint detector, so relative head pose and expression transfer across domains without adapters; an expression consistency loss from a pretrained AffectNet emotion encoder improves facial dynamics.
Key results
- On cross-domain tasks, AnyTalk achieves the lowest FID in all four transfer settings, e.g., 13.039 on Disney Human→Real versus 15.224 for Face-vid2vid.
- In a user study with 25 participants, AnyTalk was chosen 59.20% of the time, versus 18.13% for ToonTalker and 3.20% for FOMM.
- AnyTalk runs at 42 fps with naive PyTorch, compared with 17 fps for ToonTalker.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AnyTalk instead of Face-vid2vid | Cross-domain face reenactment · Disney Human → Real | FID: 15.224 -> 13.039 | Helped |
| use AnyTalk instead of DaGAN | Cross-domain face reenactment · Real → Disney Human | FID: 23.665 -> 24.144 | Hurt |
| use AnyTalk instead of DaGAN | Cross-domain face reenactment · Real → Disney Human | CSIM: 0.821 -> 0.830 | Helped |
| use AnyTalk instead of Face-vid2vid | Cross-domain face reenactment · Real → Disney Human | CEIM: 0.8899 -> 0.8936 | Helped |
| use AnyTalk instead of Face-vid2vid | Cross-domain face reenactment · Disney Animals → Real | FID: 15.977 -> 13.312 | Helped |
| use AnyTalk instead of ToonTalker | Cross-domain face reenactment · Disney Animals → Real | CSIM: 0.835 -> 0.848 | Helped |
| use AnyTalk instead of ToonTalker | Cross-domain face reenactment · Disney Animals → Real | CEIM: 0.9047 -> 0.9065 | Helped |
| use AnyTalk instead of DaGAN | Cross-domain face reenactment · Real → Disney Animals | FID: 45.531 -> 44.481 | Helped |
| use AnyTalk instead of DaGAN | Cross-domain face reenactment · Real → Disney Animals | CSIM: 0.768 -> 0.770 | Helped |
| use AnyTalk instead of Face-vid2vid | Cross-domain face reenactment · Real → Disney Animals | CEIM: 0.9131 -> 0.9139 | Helped |
| add expression consistency loss Lexp (ablation) | AnyTalk · Disney Human → Real | CEIM: 0.8923 -> 0.8927 | Helped |
| add expression consistency loss Lexp (ablation) | AnyTalk · Real → Disney Human | CEIM: 0.8899 -> 0.8936 | Helped |
Surprising findings
- On Real→Disney Human, AnyTalk's FID (24.144) is slightly worse than Face-vid2vid (24.921)?
Limitations
- Relies on a pretrained expression encoder and sparse keypoints, which the authors note are hard for recognizing fine expressions
- Audio-driven and text-driven support goes through mesh videos as an intermediary rather than direct audio conditioning
Experimental setup
Datasets: VoxCeleb1 AniTalk AffectNet CUFS
Benchmarks: AniTalk VoxCeleb1 CUFS
Metrics: FID CSIM CEIM NIQE LSE-D user preference rate
Baselines: FOMM DaGAN ToonTalker Face-vid2vid SadTalker
Builds on: Face-vid2vid First Order Motion Model DaGAN ToonTalker
Closest papers by meaning
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.89)
- Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (NeurIPS 2025 · similarity 0.89)
- VQTalker: Towards Multilingual Talking Avatars Through Facial Motion Tokenization (AAAI 2025 · similarity 0.88)
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.