Research map / Video / Video generation
From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a hierarchical visual encoder (content, timbre, prosody stages aligned with lips, face identity, facial expressions) feeding a flow matching mel-spectrogram decoder for high-quality video-to-speech synthesis.
Problem
Video-to-speech synthesis from silent talking-face video suffers from a large modality gap between silent video and speech, leaving generated speech quality far below real utterances.
Approach
Gradually transform silent video into acoustic feature spaces in three ordered stages (content from lip motion, timbre from face identity, prosody from facial expressions), each aligned to a corresponding speech attribute, and decode mel-spectrograms with an OT conditional flow matching U-Net decoder conditioned on the resulting visual encoding.
Key results
- Naturalness MOS of 4.49 vs 4.54 for ground truth (gap 0.05), far above DiffV2S (2.97) and LTBS (2.52) on LRS3.
- UTMOS 4.031 on LRS3-TED with 10 sampling steps, versus 3.058 for DiffV2S (1000 steps) and 3.545 for ground truth.
- Speaker similarity (GE2E 0.650, VoxSim 0.495) is best among video-driven speaker methods; removing hierarchical modeling raises WER from 30.45 to 33.64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical content, timbre, and prosody modeling instead of modeling acoustic attributes simultaneously (ablation) | Proposed VTS system · LRS3 test | UTMOS: 3.737 -> 4.031 | Helped |
| Add timbre modeling and its mapper (ablation) | Proposed VTS system · LRS3 test | UTMOS: 3.858 -> 4.031 | Helped |
| Add prosody modeling and its mapper (ablation) | Proposed VTS system · LRS3 test | UTMOS: 3.866 -> 4.031 | Helped |
| Add facial identity features (ablation) | Proposed VTS system · LRS3 test | UTMOS: 4.001 -> 4.031 | Helped |
| Add facial expression features (ablation) | Proposed VTS system · LRS3 test | UTMOS: 3.965 -> 4.031 | Helped |
| Add weighted summation of AV-HuBERT intermediate features (ablation) | Proposed VTS system · LRS3 test | UTMOS: 4.038 -> 4.031 | No clear effect |
| Add masked convolution blocks to content and prosody predictors (ablation) | Proposed VTS system · LRS3 test | UTMOS: 3.986 -> 4.031 | Helped |
| Apply classifier-free guidance (ablation) | Proposed VTS system · LRS3 validation | UTMOS: 3.799 -> 3.946 | Helped |
| Increase sampling steps from 10 to 1000 | Proposed VTS system · LRS3-TED test | UTMOS: 4.031 -> 3.993 | No clear effect |
Surprising findings
- Generated speech scores higher than ground-truth audio on UTMOS and DNSMOS, attributed to the absence of background noise in the synthesis.
- Using 10 versus 1000 sampling steps yields nearly identical quality, showing the flow matching decoder needs few steps.
Limitations
- Evaluated only on LRS3 and LRS2 in-the-wild datasets, both English TED/BBC content.
- Teacher-forcing is used to train the acoustic attribute adapters, which may create a train-inference mismatch.
- Ground-truth WER and RMSEf0 remain far from the synthesized values, indicating remaining intelligibility gap.
Experimental setup
Datasets: LRS3-TED LRS2-BBC LJ Speech
Benchmarks: LRS3-TED LRS2-BBC
Metrics: MOS naturalness MOS intelligibility UTMOS DNSMOS RMSEf0 WER SECS (GE2E, VoxSim) MAEE
Baselines: SVTS Intelligible LTBS DiffV2S
Builds on: flow matching Matcha-TTS AV-HuBERT DiffV2S Intelligible
Closest papers by meaning
- Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.90)
- Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · similarity 0.89)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.89)
- ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.88)
- TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (NeurIPS 2025 · similarity 0.88)
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.