Research map / Video / Video segmentation and tracking
Restoring Speaking Lips from Occlusion for Audio-Visual Speech Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVLR, a framework that detects occluded lip frames, masks occlusions via CAM, and restores lips with a synthesis-matching audio-visual inpainting network so existing AVSR models can be used without retraining.
Problem
Audio-visual speech recognition degrades severely when lips are visually occluded (by a microphone or hand) in real-world videos, and existing AVSR models assume visible lips.
Approach
Restore occluded lips before recognition: CAM from an occluded-frame detector guides masking, and a synthesis-matching auto-encoder pair uses noisy audio plus an unoccluded appearance cue to synthesize then refine the lower face, supervised by a frozen AV-Hubert lip-reading loss and an audio-visual contrastive loss.
Key results
- AV-Hubert WER at -5dB rises from 10.62% (clean) to 13.87% under lip occlusion and recovers to 11.87% with AVLR on LRS2.
- AVLR reduces occlusion-induced degradation by 71.4% for TM-CTC and 75% for Conformer on LRS2 at -5dB.
- Synthesis-matching achieves 29.87% WER versus 32.39% for TalkLip and 35.26% for Wav2lip in the talking-face comparison ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Restore lip-occluded video with AVLR | AV-Hubert · LRS2, 10dB to -5dB | WER: 3.48 -> 3.45; 3.95 -> 3.87; 6.58 -> 5.69; 13.87 -> 11.87 | Helped |
| Restore lip-occluded video with AVLR | AV-Hubert · LRS3, 10dB to -5dB | WER: 1.8 -> 1.9; 2.5 -> 2.4; 5.2 -> 4.8; 15.2 -> 13.3 | Mixed |
| Restore lip-occluded video with AVLR | TM-CTC · LRS2, 10dB to -5dB | WER: 16.4 -> 13.9; 25.5 -> 19.4; 42.7 -> 31.7; 66.7 -> 52.5 | Helped |
| Restore lip-occluded video with AVLR | P&U net · LRS2, 10dB to -5dB | WER: 6.7 -> 6.1; 9.2 -> 8.1; 16.3 -> 13.2; 32.6 -> 25.0 | Helped |
| Restore lip-occluded video with AVLR | Conformer · LRS2, 10dB to -5dB | WER: 8.9 -> 8.4; 13.1 -> 11.6; 25.8 -> 20.5; 48.7 -> 37.2 | Helped |
| Restore lip-occluded video with AVLR | P&U net · LRS3, 10dB to -5dB | WER: 5.7 -> 4.4; 9.1 -> 6.3; 18.3 -> 12.1; 42.1 -> 31.1 | Helped |
| Restore lip-occluded video with AVLR | Conformer · LRS3, 10dB to -5dB | WER: 6.7 -> 6.2; 10.9 -> 9.8; 26.7 -> 22.5; 56.8 -> 46.5 | Helped |
| Add contrastive loss (ablation) | AVLR · LRS2 | WER%: 29.87 -> 31.70 | Helped |
| Add lip-reading loss (ablation) | AVLR without contrastive loss · LRS2 | WER%: 31.70 -> 33.77 | Helped |
Surprising findings
- Talking-face generation methods such as Wav2lip and TalkLip were not clearly better than the proposed synthesis-matching approach on visual quality metrics, with PSNR lower than Wav2lip.
Limitations
- At -5 dB noise, a gap between original and recovered-video WER remains for all evaluated AVSR methods.
- Occlusions are simulated by pasting objects onto frames rather than using naturally occluded real-world footage.
- Lip-reading expert is fine-tuned without affine preprocessing, so AV-Hubert results differ from originally reported numbers.
Experimental setup
Datasets: LRS2 Naturalistic Occlusion Generation dataset
Benchmarks: LRS2 LRS3
Metrics: WER SSIM PSNR LSE-C LSE-D SI-SDRi
Baselines: TM-CTC Conformer P&U net Wav2lip TalkLip
Builds on: CAM AV-Hubert Talklip infoNCE
Closest papers by meaning
- Visual Hallucination Elevates Speech Recognition (AAAI 2024 · similarity 0.85)
- Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.85)
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.85)
- Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation (ICLR 2025 · similarity 0.85)
- OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers (NeurIPS 2025 · similarity 0.85)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.