Research map / Vision-language and multimodal / Cross-modal learning

VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual speech separation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VSRo-200, a 200-hour Romanian podcast lip-reading dataset with pseudo-labels (full) and human transcripts (100 h), and benchmarks supervision quality, OOD robustness, confidence-aware audio-visual fusion, and transfer to LRRo.

Problem

Visual speech recognition (lip reading) for Romanian is under-resourced: existing datasets are small word-level sets, and large pseudo-labeled corpora do not isolate the effect of annotation quality, OOD robustness, or multimodal fusion in low-resource settings.

Approach

A Romanian sentence-level VSR dataset with paired human and pseudo-label supervision on overlapping data, enabling controlled comparison of supervision quality, plus a confidence-aware (entropy-weighted) token-level audio-visual fusion between Whisper and a visual model.

Key results

• Human-annotated 100 h reaches 53.3% WER vs 53.6% for pseudo-labels at 100 h on unseen speakers; pseudo-labels scaled to 200 h reduce WER to 48.8%.
• Under -5 dB Gaussian noise, fine-tuned Whisper WER drops from 73.49% to 38.87% with audio-visual fusion.
• On LRRo, best MultiVSR+MLP reaches 95.0% top-1 on Lab and 72.7% top-1 on Wild, versus 51.6% prior best on Wild.

Surprising findings

• Fine-tuned Whisper under babble noise at -5 dB reaches 137.05% WER, exceeding 100% due to insertions.
• Visual-only model is noise-independent, yet fused fine-tuned model gains are largest where audio is weakest.

Limitations

• Evaluation limited to a single architecture (MultiVSR) and few training runs.
• Podcast data has consistent visual conditions, limiting generalization to unconstrained settings.
• Pseudo-label noise and OOV words limit performance in domain-specific settings.

Experimental setup

Datasets: VSRo-200 LRRo Romanian podcast OOD test sets

Benchmarks: VSRo-200 unseen/seen speaker test sets VSRo-200 OOD categories LRRo Lab LRRo Wild

Metrics: WER CER top-1 accuracy top-5 accuracy OOV token rate OOV type rate

Baselines: Whisper zero-shot Whisper fine-tuned Inception-V4 ResNet + BiGRU + LI ResNet + MS-TCN + LI

Builds on: MultiVSR Whisper SyncNet Pyannote

Closest papers by meaning

• Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.88)
• $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time (NeurIPS 2025 · similarity 0.88)
• MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition (NeurIPS 2025 · similarity 0.87)
• Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation (ICLR 2025 · similarity 0.86)
• ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.86)
• MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition (ICML 2025 · similarity 0.86)