Research map / Vision-language and multimodal / Cross-modal learning
Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAV2vec, a self-distillation audio-visual speech representation learning method that predicts clean teacher targets from corrupted inputs, using unimodal cross-modal corrupted prediction (ACP/VCP) to improve robustness of AVSR under joint audio-visual corruption.
Problem
Audio-visual speech recognition degrades under joint audio and visual corruptions (e.g., lip occlusion, blur, background noise), and prior methods rely on specialized architectures or external modules.
Approach
Self-distillation with a corrupted prediction task where the student sees corrupted input and predicts clean teacher targets; unimodal multi-task learning predicts clean audio targets from corrupted video only and clean video targets from corrupted audio only, aligning corrupted modalities without architectural changes.
Key results
- On LRS3 with object occlusion + MUSAN noise, CAV2vec achieves average N-WER 5.1% versus 6.2% for AV-data2vec and 5.9% for AV-RelScore.
- On unseen hands occlusion and pixelated face, CAV2vec reaches N-WER 5.2% and 5.1%, while AV-data2vec reaches 6.5% under hands occlusion.
- On LRS2 with object occlusion and MUSAN noise, CAV2vec achieves N-WER 5.0% versus 6.4% for AV-data2vec (average N-WER across noise types).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CAV2vec corrupted prediction pretraining | AV-HuBERT-Large · LRS3, object occlusion + noise | N-WER AVG: 6.2 -> 5.1 | Helped |
| add CAV2vec corrupted prediction pretraining | AV-HuBERT-Large · LRS3, hands occlusion | N-WER AVG: 6.5 -> 5.2 | Helped |
| add CAV2vec corrupted prediction pretraining | AV-HuBERT-Large · LRS2, object occlusion + noise | N-WER AVG: 6.8 -> 5.8 | Helped |
| add corrupted prediction (AVCP) (ablation) | CAV2vec with masked prediction · LRS3, object occlusion + noise (O/MS) | N-WER: 6.2 -> 5.3 | Helped |
| use multimodal inputs with unimodal targets (mACP + mVCP) instead of AVCP (ablation) | CAV2vec · LRS3, object occlusion + noise (O/MS) | N-WER: 5.3 -> 5.1 | Helped |
| use unimodal inputs for cross-modal ACP + VCP instead of mACP + mVCP (ablation) | CAV2vec · LRS3, hands occlusion (H/MS) | N-WER: 5.4 -> 5.2 | Helped |
| use within-modal targets instead of cross-modal targets (ablation) | CAV2vec, ACP + VCP · LRS3, object occlusion + noise (O/MS) | N-WER: 5.1 -> 5.2 | Hurt |
| add AVCP to ACP + VCP (ablation) | CAV2vec · LRS3, object occlusion + noise (O/MS) | N-WER: 5.1 -> 5.2 | Hurt |
| add within-modal ACP and VCP tasks to ACP + VCP (ablation) | CAV2vec · LRS3, object occlusion + noise (O/MS) | N-WER: 5.1 -> 5.1 | No clear effect |
| increase audio and visual corruption ratios to 0.5 and 0.7 (ablation) | CAV2vec · LRS3, object occlusion + noise | N-WER AVG: 5.1 -> 4.7 | Helped |
| increase corruption exposure (ablation) | CAV2vec · LRS3, clean | WER: 1.5 -> 1.6 | Hurt |
Surprising findings
- Unimodal cross-modal corrupted prediction (ACP+VCP) outperforms multimodal-input variants (mACP+mVCP) and within-modal variants.
- RAVEn and BRAVEn, which use separate unimodal encoders, perform poorly in severely corrupted audio conditions despite very large parameter counts (673M).
Limitations
- Relies on initialization from an existing AV-HuBERT-Large checkpoint pretrained on noise-augmented LRS3 and VoxCeleb2, so it is not trained from scratch.
- Evaluated mainly on LRS3 and LRS2 English AVSR with simulated corruptions; real-world corruption not directly tested.
- Corruption ratios trade off performance between noisy and clean conditions.
Experimental setup
Datasets: LRS3 LRS2 VoxCeleb2 MUSAN COCO DEMAND 11k-Hands
Benchmarks: LRS3 LRS2 DEMAND noise evaluation
Metrics: WER N-WER
Baselines: V-CAFE RAVEn BRAVEn AV-HuBERT AV-data2vec AV-RelScore
Builds on: AV-HuBERT AV-data2vec data2vec modality dropout
Closest papers by meaning
- $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time (NeurIPS 2025 · similarity 0.89)
- Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation (AAAI 2024 · similarity 0.87)
- Visual Hallucination Elevates Speech Recognition (AAAI 2024 · similarity 0.87)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.86)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.86)
- MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.