Research map / Vision-language and multimodal / Cross-modal learning
Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AV-wav2vec2, a multichannel audio-visual self-supervised pre-training framework using intra- and inter-channel contrastive losses plus extra single-channel audio, evaluated on Chinese MISP2021-AVSR for AVSR/ASR/VSR and AVSD.
Problem
Self-supervised speech pre-training has been explored for near-field single-channel speech, but far-field multichannel and multi-modal (audio-visual) speech representation learning is under-explored, and labeled multichannel data is scarce.
Approach
A multi-path audio-visual encoder fuses video features with per-channel audio features in a shared transformer; contrastive targets come from each microphone channel (intra- and inter-channel losses), with single-channel unlabeled audio added via joint contrastive training, avoiding multi-iteration clustering or mean-teacher methods.
Key results
- Multichannel AV-wav2vec2-CTC pre-trained 200k steps reaches far-field AVSR CER 43.74/70.38/75.6/79.57 on C1-C4, improving over AV-HuBERT-CTC (62.86/79.99/82.33/85.72) on the far-field setting.
- Adding 900 hours of WenetSpeech unlabeled audio lowers the multichannel far-field AVSR CER to 41.91/68.24/73.27/77.82 on C1-C4.
- On AVSD, AV-wav2vec2 visual+audio replacement reduces test DER from a 13.58 baseline to 10.82, versus 11.96 for AV-HuBERT visual+audio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 900 hours of unlabeled single-channel audio for joint training (ablation) | AV-wav2vec2-CTC · MISP2021-AVSR far-field C4 | CER (%): 71.03 -> 68.24 | Helped |
| Add inter-channel contrastive loss (ablation) | AV-wav2vec2 · MISP2021-AVSR C4 far-field AVSR | CER (%): 73.12 -> 70.65 | Helped |
| Use AV-wav2vec2 pre-training instead of MS-TCN | Far-field ASR · MISP2021-AVSR C4 | CER (%): 84.21 -> 77.82 | Helped |
| Use AV-wav2vec2 pre-training instead of MS-TCN | Far-field VSR · MISP2021-AVSR C4 | CER (%): 95.69 -> 84.21 | Helped |
| Use AV-wav2vec2 pre-training instead of MS-TCN | Far-field AVSR · MISP2021-AVSR C4 | CER (%): 73.73 -> 69.47 | Helped |
| Replace both AVSD baseline branches with AV-wav2vec2 features | AVSD · MISP2021-AVSD Eval | DER: 13.58 -> 10.82 | Helped |
| Replace both AVSD baseline branches with VATLM features | AVSD · MISP2021-AVSD Eval | DER: 13.58 -> 11.23 | Helped |
Surprising findings
- The Chinese AV-HuBERT-ED with 4000 modeling units failed to converge (CER near 100%), requiring a CTC head instead.
- Adding extra 900 hours of single-channel data gave only modest overall gains, attributed to the dialect nature of MISP2021-AVSR data.
Limitations
- Pre-training and evaluation are restricted to a single Chinese dataset recorded in home TV scenarios, mainly in a local dialect.
- Pre-trained AV-HuBERT and VATLM baselines are English models, causing domain mismatch.
- Gains from extra single-channel data are small overall.
Experimental setup
Datasets: MISP2021-AVSR WenetSpeech
Benchmarks: MISP2021-AVSR
Metrics: CER DER
Baselines: MS-TCN AV-HuBERT-CTC AV-HuBERT-ED VATLM AVSD baseline (Chen et al.)
Builds on: wav2vec 2.0 AV-HuBERT MS-TCN BeamformIt
Closest papers by meaning
- Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation (ICLR 2025 · similarity 0.87)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.87)
- ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.84)
- video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.84)
- AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation (CVPR 2024 · similarity 0.84)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.