Research map / Large language models / Language, safety and interpretability
Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a speaker-adaptive lip reading method that adapts a pre-trained LLM-based lip reader at both vision (padding prompts + LoRA in visual encoder) and language (input prompt tuning in the LLM decoder) levels, and introduces the VoxLRS-SA sentence-level in-the-wild English adaptation dataset.
Problem
Lip reading models degrade on unseen speakers, and prior speaker adaptation works only adapt visual features on small word-level datasets (GRID, LRW-ID) that lack pose diversity and vocabulary size; language-level speaker adaptation is unexplored for lip reading.
Approach
Combine padding prompts and LoRA in the visual encoder (front-end spatial, back-end temporal) with input prompt tuning on a frozen LLM decoder, so that both lip appearance/movement and speaker-specific language patterns are adapted with under 1M trainable parameters.
Key results
- Vision-level adaptation with the proposed method reaches 41.5% WER with +0.7M trainable encoder parameters vs. 47.3% baseline, outperforming V LoRA (42.5%) and Padding Prompt (42.9%).
- Language-level input prompt tuning (0.04M params) achieves 44.1% WER vs. 44.7% for LoRA on the LLM (4.7M params).
- Adaptation with 5 minutes of speaker data reduces mean WER from 47.3% to 43.4%; with 45 minutes, 40.9%. Finetune-F&B with language prompts reaches the best 39.6%, slightly better than the proposed 40.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt the visual encoder with padding prompts and LoRA | LLaMA3-8B lip-reading model · VoxLRS-SA | WER (%): 47.3 -> 41.5 | Helped |
| Fine-tune the visual encoder front- and back-ends and add an input prompt | LLaMA3-8B lip-reading model · VoxLRS-SA | WER (%): 47.3 -> 39.6 | Helped |
| Combine visual adaptation and language-level input prompt tuning | LLaMA3-8B lip-reading model · VoxLRS-SA | WER (%): 47.3 -> 40.9 | Helped |
| Use input prompt tuning instead of LLM LoRA for language-level adaptation | LLaMA3-8B lip-reading model · VoxLRS-SA | WER (%): 44.7 -> 44.1 | Helped |
| Use 1 minute of speaker-specific adaptation data | Proposed speaker-adaptive lip-reading model · VoxLRS-SA | Mean WER (%): 47.3 -> 46.2 | Helped |
| Increase adaptation data from 1 to 5 minutes | Proposed speaker-adaptive lip-reading model · VoxLRS-SA | Mean WER (%): 46.2 -> 43.4 | Helped |
| Increase adaptation data from 5 to 15, 30, and 45 minutes | Proposed speaker-adaptive lip-reading model · VoxLRS-SA | Mean WER (%): 43.4 -> 42.5 -> 41.4 -> 40.9 | Helped |
| Apply LoRA to convolution, query, key, and value weights rather than query, key, and value only (ablation) | Vision-level adaptation on VoxLRS-SA S1 · VoxLRS-SA, S1 | WER (%): 38.5 -> 38.6 | Hurt |
| Apply LoRA to value weights only (ablation) | Vision-level adaptation · VoxLRS-SA, S1 | WER (%): 42.2 -> 39.6 | Helped |
| Apply LoRA to query, key, and value weights (ablation) | Vision-level adaptation · VoxLRS-SA, S1 | WER (%): 42.2 -> 38.5 | Helped |
| Use rank 4 rather than rank 8 for LoRA on convolution, query, key, and value weights (ablation) | Vision-level adaptation · VoxLRS-SA, S1 | WER (%): 38.6 -> 38.9 | Hurt |
| Use rank 2 rather than rank 8 for LoRA on convolution, query, key, and value weights (ablation) | Vision-level adaptation · VoxLRS-SA, S1 | WER (%): 38.6 -> 38.4 | Helped |
Surprising findings
- Full fine-tuning of the whole visual encoder (Finetune-F&B, 181.3M params) gives a lower WER (39.6%) than the proposed parameter-efficient method (40.9%) in the combined setting.
- With only 1 minute of adaptation data, the method improves the mean WER only by 1.2 points and some speakers get worse than the baseline.
Limitations
- Adaptation still requires training and speaker-specific data, which is hard to collect at scale.
- Proposed combined method does not beat full fine-tuning of the visual encoder in WER.
- Some speakers worsen with 1 minute of adaptation data; failure cases show context misinterpretation and word substitutions.
Experimental setup
Datasets: VoxLRS-SA VoxCeleb2 LRS3 GRID LRW-ID
Benchmarks: VoxLRS-SA
Metrics: WER
Baselines: LoRA (vision) LoRA (language) Padding Prompt Finetune-F Finetune-B Finetune-F&B
Builds on: LoRA padding prompt input prompt tuning Conformer LLM-based lip reading
Closest papers by meaning
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.89)
- Language Without Borders: A Dataset and Benchmark for Code-Switching Lip Reading (NeurIPS 2024 · similarity 0.88)
- MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading (AAAI 2025 · similarity 0.85)
- Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.85)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.85)
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.