Research map / Label-efficient and robust learning / Self-supervised and contrastive
Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a single Transformer encoder-decoder (USR) for ASR, VSR and AVSR jointly, combining self-supervised pre-training with greedy-pseudo-label semi-supervised fine-tuning, reaching SOTA on LRS3, LRS2 and WildVSR.
Problem
Speech recognition for auditory, visual and audiovisual inputs is usually handled by separate models; existing unified self-supervised approaches need task-specific fine-tuning, overfit with few labels, and VSR is hard to optimise from scratch.
Approach
A single shared encoder-decoder trained jointly on audio, video and audiovisual inputs (weight sharing amortises feature extraction and pseudo-label cost), with an EMA teacher generating greedy CTC and attention pseudo-labels filtered by token-wise confidence, plus an audiovisual-target masked prediction pre-training stage.
Key results
- Best USR model (Large, LRS3+Vox2, high-resource) reaches 22.3% WER for VSR, 1.2% ASR and 1.1% AVSR on LRS3, outperforming u-HuBERT (29.1% VSR).
- Low-resource (30 h labelled) Large model reaches 26.9% VSR and 2.4% ASR/AVSR WER, surpassing BRAVEn on VSR.
- On LRS3 with shallow-fusion LM, USR reaches 21.5% VSR WER, surpassing Auto-AVSR's 23.5% with a single shared model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Share model parameters across modalities rather than train separate models (ablation) | Unified supervised training · LRS3 full-data, VSR and AVSR | WER (%): VSR: 85.5 -> 36.4; AVSR: 63.4 -> 2.1 | Helped |
| Increase the visual loss weight from 0.3 to 0.5 (ablation) | Unified supervised training · LRS3 full-data, VSR versus ASR/AVSR | WER (%): VSR: 36.4 -> 35.2; ASR: 2.3 -> 2.4; AVSR: 2.1 -> 2.2 | Mixed |
| Set separate labelled-loss weights with gamma_a=0.5 and gamma_v=0.2 (ablation) | Unified semi-supervised training · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 42.3 -> 37.8; ASR: 4.1 -> 4.0; AVSR: 4.0 -> 3.9 | Helped |
| Use EMA teacher updates instead of copying student weights each iteration (ablation) | Unified semi-supervised training · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 38.9 -> 37.8; ASR: 4.1 -> 4.0; AVSR: 4.0 -> 3.9 | Helped |
| Use CTC-attention losses instead of CTC alone (ablation) | Unified semi-supervised training · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 45.6 -> 37.8; ASR: 5.2 -> 4.0; AVSR: 5.0 -> 3.9 | Helped |
| Add self-supervised pre-training before semi-supervised fine-tuning (ablation) | Unified Base model · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 37.8 -> 36.0; ASR: 4.0 -> 3.2; AVSR: 3.9 -> 3.0 | Helped |
| Use audiovisual rather than audio-only pre-training targets (ablation) | Unified self-supervised pre-training · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 37.3 -> 36.0; ASR: 3.2 -> 3.2; AVSR: 3.1 -> 3.0 | Mixed |
| Average encoder-block targets instead of using only the last block (ablation) | Unified self-supervised pre-training · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 37.2 -> 36.0; ASR: 3.4 -> 3.2; AVSR: 3.1 -> 3.0 | Helped |
| Increase predictor depth from 2 to 4 (ablation) | Unified self-supervised pre-training · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 36.0 -> 36.9; ASR: 3.2 -> 3.1; AVSR: 3.0 -> 2.9 | Mixed |
| Use soft sampling instead of hard greedy sampling for attention pseudo-labels (ablation) | Unified semi-supervised training · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 36.0 -> 37.5; ASR: 3.2 -> 3.4; AVSR: 3.0 -> 3.4 | Hurt |
| Increase pre-training mask probability from 0.4 to 0.8 (ablation) | Unified self-supervised pre-training · LRS3 low-resource, VSR/ASR/AVSR | WER (%): VSR: 36.0 -> 38.0; ASR: 3.2 -> 3.3; AVSR: 3.0 -> 3.1 | Hurt |
| Add supervised fine-tuning tricks to semi-supervised fine-tuning (ablation) | Unified semi-supervised fine-tuning · LRS3 low-resource, VSR | WER (%): 36.0 -> 39.3 | Hurt |
Surprising findings
- Training VSR and AVSR from scratch with weight sharing across modalities avoids the optimisation difficulties typically requiring self-supervised pre-training or curriculum learning.
- Semi-supervised fine-tuning is hurt by supervised fine-tuning tricks (smaller decoder, layer-wise LR decay), and audiovisual pre-training targets perform best for semi-supervised but worst-ish for supervised fine-tuning.
Limitations
- Semi-supervised fine-tuning on unlabelled data is more computationally intensive than supervised fine-tuning.
- Pseudo-label filtering is a simple threshold; alternative filtering and higher-quality labels (beam search, LM) are not explored.
- Experiments are limited to LRS3/LRS2 and a single encoder architecture; soft sampling and offline pseudo-labelling are left to future work.
Experimental setup
Datasets: LRS3 LRS2 VoxCeleb2 WildVSR
Benchmarks: LRS3 LRS2 WildVSR
Metrics: word error rate (WER)
Baselines: AV-HuBERT VATLM RAVEn AV-data2vec Lip2Vec BRAVEn u-HuBERT AV-CPL Auto-AVSR
Builds on: AV-HuBERT RAVEn BRAVEn FixMatch u-HuBERT
Closest papers by meaning
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.90)
- Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models (NeurIPS 2024 · similarity 0.89)
- Enhancing Audiovisual Speech Recognition Through Bifocal Preference Optimization (AAAI 2025 · similarity 0.89)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.88)
- Zipformer: A faster and better encoder for automatic speech recognition (ICLR 2024 · similarity 0.88)
- UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.