Research map / Image generation / Autoregressive and latent generation
Faces that Speak: Jointly Synthesising Talking Face and Speech from Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TTSF, a unified framework that jointly generates a talking face video and speech from a single portrait and text, using an OT-CFM motion sampler and motion-removed identity features to condition TTS.
Problem
Text-driven talking face generation is usually done in cascade with TTS, causing error accumulation and latency; face-stylised TTS also suffers voice inconsistency when facial motion varies for the same identity.
Approach
Jointly trains a TFG generator (LIA-based, with audio-mapper features from Matcha-TTS) and a Matcha-TTS model that is conditioned on motion-removed identity features from the TFG motion extractor; motion is sampled via OT-CFM over an autoencoder-compressed motion latent.
Key results
- On LRS2 one-shot, TTSF achieves FID 18.348 and ID-SIM 0.864, beating cascaded MakeItTalk, Audio2Head and SadTalker (best baseline FID 20.729 with TTS audio).
- In TTS voice-similarity evaluation, the w/o-motion condition reaches C-SIM 0.593 vs 0.272 for Face-TTS, and WER 14.56 vs 18.02.
- User study (40 participants, MOS) gives TTSF lip sync 4.09 vs 2.78 for the best baseline SadTalker.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| unify text-driven talking-face generation and face-conditioned speech synthesis | TTSF versus cascade baselines · LRS2 | FID: 20.729 -> 18.348 | Helped |
| unify text-driven talking-face generation and face-conditioned speech synthesis | TTSF versus cascade baselines · LRS2 | ID-SIM: 0.859 -> 0.864 | Helped |
| use motion-removed identity features instead of source image features including motion (ablation) | TTSF speech conditioning · LRS2 | C-SIM: 0.451 -> 0.593 | Helped |
| use motion-removed identity features instead of source image features including motion (ablation) | TTSF speech conditioning · LRS2 | WER: 15.68 -> 14.56 | Helped |
| remove the energy feature from the audio mapper (ablation) | TTSF · LRS2 | LSE-C: 5.721 -> 5.555 | Hurt |
| remove energy and the upsampled text feature from the audio mapper (ablation) | TTSF · LRS2 | LSE-C: 5.721 -> 3.935 | Hurt |
| add the upsampled text feature to the audio mapper (ablation) | TTSF without energy · LRS2 | LSE-C: 3.935 -> 5.555 | Helped |
| add the energy feature to the audio mapper (ablation) | TTSF without energy · LRS2 | LSE-C: 5.555 -> 5.721 | Helped |
| add the energy feature to the audio mapper (ablation) | TTSF without energy · LRS2 | DIV: 0.143 -> 0.143 | No clear effect |
| add the energy and upsampled text features to the audio mapper (ablation) | TTSF without energy and upsampled text feature · LRS2 | ID-SIM: 0.872 -> 0.876 | Helped |
Surprising findings
- Directly regressing motion codes with OT-CFM produced shaky motions, motivating the autoencoder motion normaliser.
- Including motion features in the TTS speaker condition significantly degraded voice similarity (C-SIM 0.451 vs 0.593).
Limitations
- Lip-sync LSE-C is lower than SadTalker on VoxCeleb2 (5.721 vs 6.310), so synchronisation is not uniformly best on automatic metrics.
- Training is restricted to LRS3 (indoor TED talks), while evaluation is on more challenging outdoor data.
Experimental setup
Datasets: LRS3 LRS2 VoxCeleb2
Benchmarks: LRS2 VoxCeleb2
Metrics: FID ID-SIM LSE-C DIV (Hopenet head pose std) WER MCD C-SIM RMSE F0 MOS
Baselines: MakeItTalk Audio2Head SadTalker Face-TTS
Builds on: Matcha-TTS LIA FC-TFG SadTalker Face-TTS
Closest papers by meaning
- FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles (AAAI 2025 · similarity 0.90)
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.90)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.89)
- Emotional Face-to-Speech (ICML 2025 · similarity 0.89)
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.89)
- MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.