Research map / Image generation / Editing and style transfer
FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FaceSpeak, a portrait-driven expressive TTS that disentangles identity and emotion embeddings from face images of diverse styles and injects them into VITS2, plus a new multi-style EM2TTS dataset.
Problem
Vision-driven TTS has been limited to real human faces and entangled visual features, so it cannot use non-photorealistic portraits or flexibly combine identity and emotion from different images.
Approach
Disentangle FaRL face features into identity and emotion embeddings via identity/expression adapters, using a gradient reversal emotion classifier and vCLUB mutual-information minimization, and feed the embeddings into a VITS2 backbone; also builds the EM2TTS dataset with multi-style portraits generated by PhotoMaker and DALL-E-3.
Key results
- FaceSpeak achieved intra-domain NMOS 4.13 vs 3.94 for MM-TTS and out-of-domain NMOS 4.28 vs 3.41 for MM-TTS in MOS tests.
- On intra-domain data FaceSpeak reached MCD 3.32, speaker similarity 0.95, gender accuracy 99.40% and emotion accuracy 60.92%.
- In identity/emotion control by different images, 98.6% of speech matched the identity image and 92.1% matched the emotion image.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FaceSpeak instead of MM-TTS | portrait-conditioned TTS · Intra-domain | NMOS: 3.94 ± 0.05 -> 4.13 ± 0.04 | Helped |
| use FaceSpeak instead of MM-TTS | portrait-conditioned TTS · Intra-domain | ISMOS: 3.82 ± 0.08 -> 3.97 ± 0.07 | Helped |
| use FaceSpeak instead of MM-TTS | portrait-conditioned TTS · Intra-domain | ESMOS: 4.08 ± 0.08 -> 4.36 ± 0.05 | Helped |
| use FaceSpeak instead of VITS2 | portrait-conditioned TTS · Out-of-domain | NMOS: 3.42 ± 0.05 -> 4.28 ± 0.05 | Helped |
| use FaceSpeak instead of MM-TTS | portrait-conditioned TTS · Out-of-domain | ISMOS: 3.68 ± 0.04 -> 3.77 ± 0.09 | Helped |
| use FaceSpeak instead of MM-TTS | portrait-conditioned TTS · Out-of-domain | ESMOS: 3.91 ± 0.05 -> 3.98 ± 0.07 | Helped |
| add EM²TTS multi-style portrait images to training (ablation) | FaceSpeak · Out-of-domain multi-style portraits | NMOS: 4.31 ± 0.05 -> 4.38 ± 0.06 | Helped |
| add EM²TTS multi-style portrait images to training (ablation) | FaceSpeak · Out-of-domain multi-style portraits | ISMOS: 3.88 ± 0.09 -> 4.06 ± 0.08 | Helped |
| add EM²TTS multi-style portrait images to training (ablation) | FaceSpeak · Out-of-domain multi-style portraits | ESMOS: 4.02 ± 0.06 -> 4.47 ± 0.04 | Helped |
| add EM²TTS multi-style portrait images to training (ablation) | FaceSpeak · Out-of-domain multi-style portraits | ACC_emo: 18.34 -> 26.56 | Helped |
| add EM²TTS multi-style portrait images to training (ablation) | FaceSpeak · Out-of-domain multi-style portraits | ACC_gen: 84.24 -> 92.22 | Helped |
| combine identity from one portrait with emotion from a different portrait | FaceSpeak · Combined-portrait speech control | identity match accuracy: 98.6% | Helped |
Limitations
- Out-of-domain emotion accuracy is low (31.32%), indicating weak emotion transfer to unseen portraits.
- Hard samples where physical appearance mismatches voice are difficult for models trained on the dataset.
- Evaluation relies on small-scale listening tests with 20 volunteers and 50 samples per model.
Experimental setup
Datasets: MEAD ESD EmovDB LibriTTS Oulu-CASIA
Benchmarks: MEAD-based intra-domain test out-of-domain real portraits (Oulu-CASIA with LibriTTS text) out-of-domain multi-style virtual portraits
Metrics: NMOS ISMOS ESMOS AXY preference 7-point score MCD emotion accuracy gender accuracy speaker similarity (Resemblyzer)
Baselines: VITS2 MMTTS MM-StyleSpeech MM-TTS
Builds on: VITS2 FaRL MMTTS vCLUB PhotoMaker
Closest papers by meaning
- Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · similarity 0.90)
- Emotional Face-to-Speech (ICML 2025 · similarity 0.89)
- Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion (AAAI 2025 · similarity 0.88)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.87)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.86)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.