Research map / Image generation / Autoregressive and latent generation
Emotional Face-to-Speech
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces emotional face-to-speech (eF2S), generating emotional, identity-consistent speech directly from a face via an RVQ-codec-based discrete diffusion transformer (DEmoFace) with coarse-to-fine curriculum and enhanced predictor-free guidance.
Problem
Generating speech whose identity and emotional expression are both inferred solely from a reference face (no speech prompt), extending conventional face-to-speech which captures identity only.
Approach
A discrete diffusion transformer over multi-level RVQ speech tokens with multimodal DiT blocks (AdaLN for face identity/emotion, cross-attention for text), a coarse-to-fine curriculum over RVQ levels, and an energy-based enhanced predictor-free guidance that composes joint and per-condition scores.
Key results
- Visual-only DEmoFace achieves EmoSim 0.6965 and SpkSim 0.6679 vs Face-TTS 0.5230 and 0.1968, with MCD 6.86 vs 8.46.
- Subjective MOS naturalness 3.75 for DEmoFace vs 2.92 for EmoSpeech, a ~28% relative gain.
- Visual-only DEmoFace outperforms acoustic-guided baselines on EmoSim, SpkSim, RMSE and MCD, though Face-TTS retains better WER (17.47 vs 20.78).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DEmoFace instead of Face-TTS | Visual-guided speech generation · eF2S test | EmoSim: 0.5230 -> 0.6965 | Helped |
| Use DEmoFace instead of Face-TTS | Visual-guided speech generation · eF2S test | SpkSim: 0.1968 -> 0.6679 | Helped |
| Use DEmoFace instead of Face-TTS | Visual-guided speech generation · eF2S test | WER (%): 17.47 -> 20.78 | Hurt |
| Use DEmoFace instead of EmoSpeech | Speech naturalness user study · Speech naturalness user study | MOS_nat: 2.92±0.21 -> 3.75±0.12 | Helped |
| Remove curriculum learning (ablation) | DEmoFace · Ablation study | WER (%): 20.78 -> 27.52 | Hurt |
| Remove identity feature alignment (ablation) | DEmoFace · Ablation study | SpkSim: 0.67 -> 0.58 | Hurt |
| Remove EPFG (ablation) | DEmoFace · Ablation study | EmoSim: 0.70 -> 0.67 | Hurt |
| Increase sampling steps (ablation) | DEmoFace · Sampling-step ablation | Normalized average performance | Helped |
Surprising findings
- The visual-only DEmoFace outperforms acoustic-guided methods on several objective metrics despite using no speech prompt.
- Face-TTS achieves better WER than DEmoFace despite DEmoFace being more natural and consistent on other metrics.
Limitations
- Training data is small (31.33 hours) and lacks sufficient semantic diversity, requiring an LRS3 subset.
- Word error rate lags Face-TTS, which uses about 10x more data.
- Subjective evaluation involves only 15 participants and 10 test samples.
Experimental setup
Datasets: RAVDESS MEAD MELD-FAIR LRS3
Benchmarks: RAVDESS MEAD MELD-FAIR
Metrics: EmoSim SpkSim RMSE (F0) MCD WER MOS naturalness MOS consistency
Baselines: EmoSpeech FastSpeech2 V2C-Net HPM StyleDubber Face-TTS
Builds on: discrete diffusion models predictor-free guidance DiT Encodec SpeechT5
Closest papers by meaning
- FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles (AAAI 2025 · similarity 0.89)
- Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · similarity 0.89)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.88)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.88)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.88)
- FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.