Research map / Vision-language and multimodal / Cross-modal learning

Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Self-supervised audio-visual representation learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FLARE, a visual speech encoder pretrained with Whisper-teacher random-projection masked prediction on joint face and lip inputs, decoded by a Transformer or a Qwen-2.5-7B LLM decoder for lipreading.

Problem

Visual speech recognition (lipreading) is ambiguous and sensitive to head pose and video resolution, and existing audio-visual SSL methods use only lip ROIs and jointly learn audio and visual representations rather than focusing supervision on the visual encoder.

Approach

Distill Whisper deep audio representations into visual encoders via random-projection quantized masked prediction, using a dual-stream face and lip frontend with modality dropout, a visual-dominant modality schedule, and an enhanced Conformer context encoder.

Key results

• FLARE-LLM (342M) reaches 11.7% WER on LRS2, 13.2% on LRS3, and 34.3% on WildVSR under the high-resource setting.
• In the low-resource setting, the 117M FLARE Base model gets 29.3% WER at 30h and 24.0% at 433h on LRS3, beating prior AV-SSL baselines of comparable or larger size.
• Joint face-lip input gives the lowest WER across all tested resolutions (0.25x to 1x) and lip-only input degrades significantly with head yaw, while joint input does not.

Surprising findings

• Face input outperforms lip input on LRS3 but not on LRS2, indicating the relative advantage depends on dataset characteristics such as pose distribution.

Limitations

• Joint face-lip modeling adds a second visual branch, raising computational cost versus lip-only approaches.
• Scope limited to lipreading accuracy and robustness; transfer to other audio-visual tasks and speaker information not investigated.
• FLARE-LLM uses a 7B LLM without token compression, making inference expensive (RTF ~1.3-1.4).

Experimental setup

Datasets: LRS2 LRS3 VoxCeleb2-English AVSpeech-English AVYT-talking MUSAN WildVSR

Benchmarks: LRS2 LRS3 WildVSR

Metrics: word error rate Rank_wer

Baselines: AV-HuBERT AV-data2vec RAVEn VATLM DistillAV BRAVEn Auto-AVSR Llama-AVSR Omni-AVSR UASR-LLM LP-Conformer VTP

Builds on: BEST-RQ AV-HuBERT BRAVEn DistillAV Conformer

Closest papers by meaning

• ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
• LipFormer: High-Fidelity and Generalizable Talking Face Generation With a Pre-Learned Facial Codebook (CVPR 2023 · similarity 0.87)
• From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.87)
• SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision (CVPR 2023 · similarity 0.87)
• SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory (AAAI 2022 · similarity 0.87)
• ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.86)