Research map / Video / Video generation
LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LipVoicer, which generates mel-spectrograms from silent video with a DDPM conditioned on lip and face embeddings, and at inference steers generation toward text predicted by a pretrained lip-reader using ASR classifier guidance.
Problem
Lip-to-speech synthesis from silent videos in the wild (LRS2, LRS3) yields unintelligible speech with an unrestricted vocabulary and many speakers; existing methods have high WER and weak naturalness.
Approach
Use text from a pretrained lip-reader at inference time as a classifier-guidance signal (via a fine-tuned CTC ASR operating on noisy mel-spectrograms) on a video-conditioned diffusion model, with classifier-free guidance for video conditioning and gradient normalization for guidance stability.
Key results
- On LRS3, LipVoicer reaches 21.4% WER versus 57.4% for Lip2Speech, 82.4% for SVTS and 90.6% for VCA-GAN, with DNSMOS 3.11 vs 2.36 for Lip2Speech.
- Human MOS on LRS2 intelligibility is 3.53 for LipVoicer vs 2.07 for Lip2Speech and 1.77 for VCA-GAN (GT 4.33).
- Removing ASR guidance (w2=0) raises LRS3 WER from 21.4% to 86.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LipVoicer instead of Lip2Speech | Lip-to-speech generation · LRS2 | WER: 51.4% -> 17.8% | Helped |
| use LipVoicer instead of Lip2Speech | Lip-to-speech generation · LRS3 | WER: 57.4% -> 21.4% | Helped |
| use LipVoicer instead of Lip2Speech | Lip-to-speech generation · LRS2 | Intelligibility MOS: 2.07 ± 0.08 -> 3.53 ± 0.07 | Helped |
| use LipVoicer instead of Lip2Speech | Lip-to-speech generation · LRS3 | Synchronization MOS: 2.69 ± 0.08 -> 3.56 ± 0.07 | Helped |
| add classifier-free guidance (w1=2 rather than w1=-1) (ablation) | LipVoicer · LRS3 | WER: 30.1% -> 21.4% | Helped |
| add ASR text guidance (w2=1.5 rather than w2=0) (ablation) | LipVoicer · LRS3 | WER: 86.2% -> 21.4% | Helped |
| increase ASR guidance weight from w2=1.5 to w2=4 (ablation) | LipVoicer · LRS3 | DNSMOS: 3.11 -> 2.71 | Hurt |
| use a more accurate lip-reader (19.1% rather than 32.3% WER) (ablation) | LipVoicer · LRS3 | WER: 38.1% -> 21.4% | Helped |
| use ground-truth text rather than predicted lip-reader text (ablation) | LipVoicer · LRS3 | WER: 21.4% -> 5.4% | Helped |
| use reliable ASR guidance (ASR WER 3.6% rather than 6.1%) (ablation) | LipVoicer · LRS3 | WER: 76.3% -> 21.4% | Helped |
| add face embedding (ablation) | LipVoicer · LRS3 | WER: 22.9% -> 21.4% | Helped |
| train on LRS3+VoxCeleb2 instead of LRS3 (ablation) | LipVoicer · LRS3 | WER: 21.4% -> 20.6% | Helped |
Surprising findings
- Intrusive metrics PESQ and STOI rate a voice-converted version of speech similarly to a heavily noisy version, so the authors argue they are unsuitable for lip-to-speech evaluation.
- Guidance from ASR applied from the very first diffusion step degraded results; guidance was applied starting at a later diffusion step (t_ASR).
Limitations
- Guidance from ASR can reduce naturalness and cause synchronization lapses when predicted text diverges from ground truth.
- Performance depends on lip-reader accuracy; a less accurate lip-reader raises WER.
- Potential misuse: bad-faith actors could inject misleading text.
Experimental setup
Datasets: LRS2 LRS3 GRID VoxCeleb2
Benchmarks: LRS2 LRS3 GRID
Metrics: MOS WER STOI-Net DNSMOS LSE-C LSE-D PESQ STOI
Baselines: Lip2Speech SVTS VCA-GAN ReVISE
Builds on: DDPM classifier guidance classifier-free guidance DiffWave VisualVoice
Closest papers by meaning
- Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.90)
- Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · similarity 0.89)
- Language Without Borders: A Dataset and Benchmark for Code-Switching Lip Reading (NeurIPS 2024 · similarity 0.88)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.85)
- Restoring Speaking Lips from Occlusion for Audio-Visual Speech Recognition (AAAI 2024 · similarity 0.85)
- OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.