Research map / Vision-language and multimodal / Cross-modal learning

ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations

CVPR 2024 · Vision-language and multimodal › Cross-modal learning · approach: Self-supervised audio-visual representation learning · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ES3, a three-stage evolving Siamese self-supervised framework that learns audio-unique, shared, video-unique and synergistic speech information for audio-visual speech representations, and contributes the CAS-VSR-S101 Mandarin dataset.

Problem

Self-supervised learning of audio-visual speech representations is hampered by the asymmetry between audio and video; audio-centric methods capture mainly shared information and need costly manual balancing of modality parameters.

Approach

Reframes audio-visual speech representation learning as acquiring shared, unique and synergistic information via partial information decomposition, and learns these progressively in three stages (audio first, then video-unique with audio-visual bootstrapping, then joint audio-visual masked prediction) within an EMA Siamese masked-prediction framework with online clustering distillation.

Key results

• On LRS2-BBC, the BASE* model (46M, 223h pretraining and fine-tuning) reaches 31.4% VSR, 4.3% ASR and 3.8% AVSR WER, and the 433h-pretrained BASE model reaches 30.7% / 3.4% / 3.2%.
• On LRS3-TED high-resource, the BASE ES3 model achieves 40.3% VSR, 2.9% ASR and 2.5% AVSR WER; the LARGE model achieves 37.1% / 2.8% / 2.1%.
• On CAS-VSR-S101, ES3 Stage 3 reaches 58.1/55.6 VSR, 14.4/11.6 ASR and 13.4/11.0 AVSR CER (val/test), improving over the supervised LF-MMI baseline.

Measured effects reported in the paper

Surprising findings

• Stage 1 (audio-only) representations are already strong for ASR, and visual-only results of the earlier stage show a large gap versus stage 2, indicating the progressive design matters.
• The automatically learned temperature for cross-modal loss first falls then rises in stage 2, acting as an automatic balance between shared and unique information without manual tuning.

Limitations

• Final results rely on the simple LF-MMI criterion with a 2-gram-style decoding; the authors note that a larger external language model or attention decoders could improve results further.
• The visual encoder is a lightweight VGG-based 3D encoder, so visual-only (VSR) performance on LARGE models is about 2% behind SoTA.
• Pretraining on LRS2-BBC/LRS3-TED is relatively small-scale compared with prior works that use VoxCeleb2 or larger data.

Experimental setup

Datasets: LRS2-BBC LRS3-TED CAS-VSR-S101 VoxCeleb2-En

Benchmarks: LRS2-BBC LRS3-TED CAS-VSR-S101

Metrics: WER CER

Baselines: AV-HuBERT VATLM RAVEn AV-data2vec AV2vec Pan et al. Ma et al. E2E Conformer

Builds on: data2vec AV-HuBERT LF-MMI Partial Information Decomposition RAVEn

Closest papers by meaning

• From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.88)
• From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
• Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization (NeurIPS 2026 · similarity 0.87)
• video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.87)
• VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.86)
• Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.86)