Research map / Large language models / Language, safety and interpretability
VALLR: Visual ASR Language Model for Lip Reading
Marshall Thomas, Edward Fish, Richard Bowden
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add phoneme intermediate stage instead of end-to-end sentence prediction (ablation) | VALLR · LRS3 test | WER (%): 25.6 -> 18.7 | Improved |
| use Llama 3.2-3B instead of Llama 3.2-1B for phoneme-to-word LLM (ablation) | VALLR · LRS3 test | WER (%): 22.8 -> 18.7 | Improved |
| fine-tune Llama 3.2-1B on phoneme-word pairs vs GPT-2 Small (ablation) | VALLR · LRS3 test | WER (%): 33.9 -> 22.8 | Improved |
| initialise and freeze the CTC head with Wav2Vec2 ASR weights (ablation) | VALLR · LRS3 test | WER (%): +0.4% WER | Hurt |
| increase visual-encoder training data beyond 30 hours (ablation) | VALLR Video-to-Phoneme network · LRS3 test | WER (%) | No clear effect |
| add more text training data for the phoneme-to-sentence LLM (ablation) | VALLR LLM · LRS3 test | WER (%): 18.7 -> 17.5 | Improved |
| use VALLR with 30 labelled hours versus self-supervised methods with far more data | VALLR · LRS3 test | WER (%): 21.5 -> 18.7 vs USR | Improved |
| apply VALLR on LRS2 versus best self-supervised baselines | VALLR · LRS2 test | WER (%): 20.8 vs USR 15.4, RAVEn 17.9 | Mixed |
Closest papers by meaning
- Sub-Word Level Lip Reading With Visual Attention (CVPR 2022 · similarity 0.92)
- Deep Audio-Visual Speech Recognition (TPAMI 2018 · similarity 0.92)
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.90)
- Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · similarity 0.89)
- Language Without Borders: A Dataset and Benchmark for Code-Switching Lip Reading (NeurIPS 2024 · similarity 0.89)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
