Research map / Image generation / Autoregressive and latent generation
Let There Be Sound: Reconstructing High Quality Speech from Silent Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a lip-to-speech system that predicts HuBERT-derived linguistic units, pitch and energy, and refines mel-spectrograms with a flow-based post-net, achieving near-vocoded quality without text labels.
Problem
Lip-to-speech suffers from a one-to-many mapping caused by homophenes and diverse speech variations, producing mispronounced and over-smoothed speech.
Approach
Condition a lip-to-mel model on quantised intermediate HuBERT (layer 12, 200 K-means clusters) linguistic units plus pitch and energy predictors, and add a flow-based post-net to refine fine-grained mel details, all without text labels.
Key results
- On GRID, the proposed method reaches naturalness 4.46 and intelligibility 4.63 MOS versus 4.74/4.79 for vocoded speech, with WER 17.07 vs 12.44 for vocoded and 17.62 for VCA-GAN.
- On Lip2Wav, the method achieves naturalness 4.15 and WER 34.71 versus 48.73 for VCA-GAN and 61.09 for SVTS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the linguistic predictor (ablation) | Let There Be Sound · Lip2Wav | WER: 34.71 -> 42.51 | Helped |
| Add the pitch predictor (ablation) | Let There Be Sound · Lip2Wav | WER: 34.71 -> 39.96 | Helped |
| Add the energy predictor (ablation) | Let There Be Sound · Lip2Wav | WER: 34.71 -> 40.58 | Helped |
| Add the flow-based post-net (ablation) | Let There Be Sound · Lip2Wav | CMOS naturalness: 0.00 -> -1.48 | Helped |
| Add speaker embedding (ablation) | Let There Be Sound · Lip2Wav | WER: 34.71 -> 42.33 | Helped |
| Use HuBERT layer 12 features quantized to 200 clusters rather than the other tested layer/cluster configurations (ablation) | Linguistic predictor · GRID validation | PER: 9.73 -> 8.91; 9.59 -> 8.91 | Helped |
| Use HuBERT layer 12 features quantized to 100 clusters rather than the other tested layer/cluster configurations (ablation) | Linguistic predictor · GRID validation | WER: 18.02 -> 16.53; 17.57 -> 16.53 | Helped |
Surprising findings
- Using HuBERT layer 24 with 200 clusters gives a much higher WER (29.17) than layer 12, showing sensitivity to the layer choice.
Limitations
- Relies on a pre-trained neural vocoder (Fre-GAN) rather than an end-to-end pipeline.
- Lip2Wav quality still degrades substantially on unconstrained data (WER 34.71).
Experimental setup
Datasets: GRID Lip2Wav
Benchmarks: GRID Lip2Wav
Metrics: MOS naturalness MOS intelligibility WER CER CMOS PER pitch moments energy MAE
Baselines: VCA-GAN SVTS MT GlowLTS
Builds on: HuBERT Fre-GAN GlowTTS SVTS VCA-GAN
Closest papers by meaning
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.90)
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.90)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.86)
- TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (NeurIPS 2025 · similarity 0.86)
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.