Research map / Vision-language and multimodal / Cross-modal learning
Deep Audio-Visual Speech Recognition
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, Andrew Zisserman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add audio to video input (AV vs V) | TM-seq2seq · LRS2-BBC noisy audio | WER: 58.0% -> 35.9% | Improved |
| add video to audio input on clean audio | TM-CTC · LRS2-BBC clean | WER: 10.1% -> 8.2% | Improved |
| add video to audio input on clean audio | TM-seq2seq · LRS2-BBC clean | WER: 9.4% -> 8.5% | Improved |
| add external language model | TM-CTC · LRS2-BBC video-only | WER: 65.0% -> 54.7% | Improved |
| add external language model | TM-seq2seq · LRS2-BBC video-only | WER: 49.8% -> 48.3% | Improved |
| use seq2seq instead of CTC for lip reading | TM-seq2seq · LRS2-BBC video-only | WER: 54.7% -> 48.3% | Improved |
| use CTC instead of seq2seq under loud noise | TM-CTC · LRS2-BBC noisy audio-visual | WER: 34.2% -> 23.6% | Improved |
| train with babble noise at probability 0.25 | audio-visual models · training | WER | Improved |
| introduce curriculum learning from single words to full sentences | TM models · training | convergence | Improved |
| fine-tune on randomly shifted video | TM-seq2seq · LRS2-BBC out-of-sync | WER | Improved |
| fine-tune on randomly shifted video | TM-CTC · LRS2-BBC out-of-sync | WER | No clear effect |
| increase beam width | TM-seq2seq · LRS2-BBC video-only | WER | Improved |
Closest papers by meaning
- Lip Reading Sentences in the Wild (CVPR 2017 · similarity 0.95)
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.90)
- Language Without Borders: A Dataset and Benchmark for Code-Switching Lip Reading (NeurIPS 2024 · similarity 0.89)
- Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · similarity 0.89)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.88)
- Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
