Research map / Video / Video generation
FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates diverse, audio-consistent talking-face videos from a single audio input by progressively disentangling identity, content, and emotion with 3DMM priors and adapting frozen latent diffusion models with three trainable adapters plus autoregressive inference.
Problem
Generating high-fidelity talking faces solely from audio, requiring decoupling of identity, content and emotion from entangled audio while keeping diversity across videos and consistency within each video.
Approach
Progressive Audio Disentanglement (identity, then content, then emotion, supervised by 3DMM coefficients) feeds a Controllable Coherent Frame generator that freezes a Latent Diffusion Model and trains three adapters (Textual Inversion, Spatial Conditional, Mask-guided Blending) with autoregressive frame-by-frame inference.
Key results
- On the Table 1 comparison with CMP, FID drops from 228.05 to 25.78 and emotion similarity rises from 0.23 to 0.76.
- Against SadTalker, Ours-E reaches EACC 0.090 (vs 0.121), FID 28.37 (vs 31.62), PSNR 29.66 (vs 29.55) and SSIM 0.67 (vs 0.64).
- Wav2Lip keeps the best Sync score (5.93), and the paper attributes this to its synchronization-model supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FaceChain-ImagineID instead of CMP | Audio-to-face generation · MEAD + HDTF test sets | Shape: 0.60 -> 0.75 | Helped |
| Use FaceChain-ImagineID instead of CMP | Audio-to-face generation · MEAD + HDTF test sets | Age: 0.41 -> 0.58 | Helped |
| Use FaceChain-ImagineID instead of CMP | Audio-to-face generation · MEAD + HDTF test sets | Gender: 0.87 -> 0.93 | Helped |
| Use FaceChain-ImagineID instead of CMP | Audio-to-face generation · MEAD + HDTF test sets | Emotion: 0.23 -> 0.76 | Helped |
| Use FaceChain-ImagineID instead of CMP | Audio-to-face generation · MEAD + HDTF test sets | FID: 228.05 -> 25.78 | Helped |
| Use the emotion-stage variant rather than the content-stage variant | FaceChain-ImagineID · MEAD + HDTF test sets | EACC: 0.113 -> 0.090 | Helped |
| Use the emotion-stage variant rather than the content-stage variant | FaceChain-ImagineID · MEAD + HDTF test sets | LMD: 2.82 -> 2.67 | Helped |
| Use the emotion-stage variant rather than the content-stage variant | FaceChain-ImagineID · MEAD + HDTF test sets | Sync: 5.86 -> 5.91 | Hurt |
| Use the emotion-stage variant rather than the content-stage variant | FaceChain-ImagineID · MEAD + HDTF test sets | SSIM: 0.60 -> 0.67 | Helped |
| Use N=8 rather than N=1 pseudo-word tokens (ablation) | TIA · CelebV-HQ | Age: 0.23 -> 0.51 | Helped |
| Use N=8 rather than N=1 pseudo-word tokens (ablation) | TIA · CelebV-HQ | Emotion: 0.49 -> 0.71 | Helped |
| Use N=16 rather than N=8 pseudo-word tokens (ablation) | TIA · CelebV-HQ | Age: 0.51 -> 0.50 | Hurt |
Surprising findings
- Jointly training TIA and SCA lets facial semantics take shortcuts from the image conditions, so sequential training is needed.
- Tuning the UNet in SCA increases trainable parameters from 282.9M to 1.15G without clear visual benefit.
Limitations
- Wav2Lip remains best on the Sync metric.
- Tuning the full UNet adds parameters without clear benefit.
- Diversity and attribute editing depend on text prompts, which are only weakly controllable in video.
Experimental setup
Datasets: MEAD VoxCeleb2 HDTF CelebV-HQ
Benchmarks: MEAD HDTF
Metrics: PSNR SSIM FID LMD Sync EACC Shape similarity Age similarity Gender similarity Emotion similarity
Baselines: CMP Wav2Lip PC-AVS EAMM SadTalker
Builds on: Latent Diffusion Models T2I-Adapter DreamBooth / textual inversion DiffTalk 3DMM
Closest papers by meaning
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.89)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.89)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.89)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.88)
- Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.