Research map / Video / Video segmentation and tracking
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that video dropout in AVSR induces audio-dominant modality bias, proposes the Modality Bias Hypothesis, and introduces MDA-KD distillation plus MS-Adapters to gain robustness to missing video without losing clean-input accuracy.
Problem
AVSR systems trained with video dropout become robust to missing video frames but degrade on complete input, and they can fall behind audio-only ASR when video is missing.
Approach
Attributes dropout-induced degradation to a shift from a bimodal to a unimodal (audio-dominated) latent decision distribution, and counters it by distilling hidden representations from a teacher trained on complete audio-visual data, plus LoRA-based modality-specific adapters that switch to audio-only decisions when video is entirely missing.
Key results
- On MISP2022 with oracle diarization, the proposed AV6 reduces test CER at full video input to 21.11% vs 21.14% for the dropout baseline AV0 while improving missing-video robustness (24.94% at 1.0 missing vs 26.65%).
- With audio-only input, MS-Adapters bring CER to 24.94%, surpassing the audio-only baseline A0 (25.13%) for the first time.
- On MISP2021 utterance-level AVSR, ROVER-combined system reaches 21.53% CER, versus 24.58% for the previous SOTA USTC system.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MDA-KD with MS-Adapter | AVSR on MISP2021 · MISP2021 | CER (%): 24.58 -> 21.53 | Helped |
| Use MDA-KD with MS-Adapter | AVSR on MISP2022 · MISP2022 | cpCER (%): 29.13 -> 28.06 | Helped |
| Add MDA-KD (ablation) | AVSR initialized from AV0 on MISP2022 · MISP2022, test dropout rate 0.00 | CER (%): 21.53 -> 21.11 | Helped |
| Add MDA-KD (ablation) | AVSR initialized from AV0 on MISP2022 · MISP2022, test dropout rate 1.00 | CER (%): 25.90 -> 25.45 | Helped |
| Add MDA-KD (ablation) | AVSR initialized from AV0 on MISP2022 · MISP2022, test dropout rate 0.00 | CER (%): 21.14 -> 21.11 | No clear effect |
| Add MS-Adapter (ablation) | AV5 on MISP2022 · MISP2022, test dropout rate 1.00 | CER (%): 25.45 -> 24.94 | Helped |
| Add MS-Adapter (ablation) | AV5 on MISP2022 · MISP2022, test dropout rate 0.00 | CER (%): 21.11 -> 21.11 | No clear effect |
| Train with dropout on all training data rather than half (ablation) | AVSR initialized from A0 on MISP2022 · MISP2022, test dropout rate 0.00 | CER (%): 21.72 -> 23.26 | Hurt |
| Train with dropout on all training data rather than half (ablation) | AVSR initialized from A0 on MISP2022 · MISP2022, test dropout rate 1.00 | CER (%): 25.64 -> 25.91 | No clear effect |
| Initialize robust training from AV0 rather than A0 (ablation) | AVSR on MISP2022 · MISP2022, test dropout rate 0.00 | CER (%): 21.72 -> 21.53 | No clear effect |
| Initialize robust training from AV0 rather than A0 (ablation) | AVSR on MISP2022 · MISP2022, test dropout rate 1.00 | CER (%): 25.64 -> 25.90 | Hurt |
| Add data augmentation to the encoder rank 32 MS-Adapter (ablation) | MS-Adapter on MISP2022 · MISP2022, test dropout rate 1.00 | CER (%): 25.35 -> 25.08 | Helped |
Surprising findings
- Video dropout during training makes AVSR converge toward audio-only ASR behavior, so higher dropout rates lead to worse clean-input performance than the no-dropout baseline.
- AVSR trained with dropout can lag behind unimodal ASR when video is completely missing.
Limitations
- Experiments limited to AVSR on far-field MISP data; generalization to other multimodal tasks is not validated.
- Dropout-induced bias analysis based on five Mandarin AVSR systems and specific MISP datasets.
- Hyperparameters such as the training dropout rate and lambda were tuned on the same test-suite setting.
Experimental setup
Datasets: MISP2021 MISP2022
Benchmarks: MISP2021 MISP2022
Metrics: CER cpCER
Baselines: Cascade Utt AV Dropout Utt Dropout Utt USTC system NIO system NPU system XMU system
Builds on: Modality Venn Diagram modality dropout (Chang et al.) AV-HuBERT LoRA intermediate CTC
Closest papers by meaning
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.87)
- $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time (NeurIPS 2025 · similarity 0.86)
- Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models (AAAI 2025 · similarity 0.86)
- Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement (AAAI 2025 · similarity 0.85)
- Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation (ICLR 2025 · similarity 0.85)
- Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.