Research map / Vision-language and multimodal / Cross-modal learning
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AV2AV, a direct, textless many-to-many audio-visual speech-to-audio-visual speech translation system trained on audio-only A2A data via unified AV-HuBERT speech units, with a zero-shot speaker AV-Renderer.
Problem
Direct translation between audio-visual speech is hard because no parallel AV-to-AV speech translation dataset exists, and prior systems are cascaded (ASR, NMT, TTS, talking face) or only output audio.
Approach
Use multilingual AV-HuBERT's modality-agnostic unified AV representations, discretized into AV speech units, so a translation model can be trained on audio-only parallel data and then applied to audio, visual, or audio-visual inputs; a zero-shot AV-Renderer (HiFi-GAN with speaker embedding plus modified Wav2Lip) generates synchronized speech and face video from the units.
Key results
- On MuAViC, the single AV2AV model reaches 26.57 X-En and 31.27 En-X ASR-BLEU, comparable to the cascaded text-based systems trained on 7.5B parallel sentences.
- On LRS3-T En-Es, the method without finetuning (25.3) already exceeds AV-TranSpeech finetuned on 10hr (21.5); with 200hr finetuning it reaches 50.9 vs 45.2.
- Under -5 dB babble noise, AV input (AV2AV) gets 17.31 BLEU vs 3.64 for audio-only A2AV and 10.36 for visual-only V2AV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multilingual AV-HuBERT rather than English-trained AV-HuBERT (ablation) | AV2AV translation model · MuAViC Es-En | ASR-BLEU: 17.63 -> 26.57 | Helped |
| use multilingual AV-HuBERT rather than English-trained AV-HuBERT (ablation) | AV2AV translation model · MuAViC Fr-En | ASR-BLEU: 16.83 -> 31.27 | Helped |
| use multilingual AV-HuBERT rather than English-trained AV-HuBERT (ablation) | AV2AV translation model · MuAViC It-En | ASR-BLEU: 21.74 -> 23.24 | Helped |
| use multilingual AV-HuBERT rather than English-trained AV-HuBERT (ablation) | AV2AV translation model · MuAViC Pt-En | ASR-BLEU: 22.22 -> 24.51 | Helped |
| add zero-shot speaker modeling to the vocoder (ablation) | AV2AV vocoder · MuAViC Es-En | voice similarity (SIM): 0.043 -> 0.222 | Helped |
| add zero-shot speaker modeling to the vocoder (ablation) | AV2AV vocoder · MuAViC Es-En | ASR-BLEU: 26.38 -> 26.57 | No clear effect |
| use AV input instead of audio-only input under acoustic noise | MuAViC Es-En translation · MuAViC Es-En, SNR -5 dB | ASR-BLEU: 3.64 -> 17.31 | Helped |
| use AV input instead of audio-only input | MuAViC Es-En translation · MuAViC Es-En, clean | ASR-BLEU: 26.04 -> 26.57 | Helped |
| train the translation model with AV speech units from multilingual rather than English-only AV-HuBERT (ablation) | AV2AV · MuAViC X-En | ASR-BLEU | Helped |
| use direct AV2AV translation rather than the 4-stage cascaded baseline | MuAViC · MuAViC X-En | ASR-BLEU: 26.57 vs 28.70; 31.27 vs 29.21; 23.24 vs 24.54; 24.51 vs 26.30 | Mixed |
| use direct AV2AV translation rather than the 4-stage cascaded baseline | MuAViC · MuAViC En-X | ASR-BLEU: 27.57 vs 30.15; 23.21 vs 24.07; 16.33 vs 25.05; 12.35 vs 22.17 | Hurt |
| pre-train on A2A data and apply without AV2A finetuning | AV2AV translation model · LRS3-T En-Es, 0hr finetuning | ASR-BLEU: 25.3 vs - | Helped |
Surprising findings
- The visual-only input system (V2AV) is unaffected by acoustic noise and outperforms audio-only systems at -5 dB SNR.
Limitations
- Requires parallel data of audio-only translation and TTS-synthesized targets for MuAViC; no native AV2AV corpus available.
- Speaker similarity and visual quality rely on Wav2Lip-based rendering, which may not faithfully model lip movements.
- Evaluation of naturalness and realness relies on MOS tests reported only in supplementary material.
Experimental setup
Datasets: LRS2 LRS3 VoxCeleb2 mTEDx AVSpeech VoxPopuli LRS3-T MuAViC MUSAN
Benchmarks: LRS3-T MuAViC
Metrics: ASR-BLEU FID LSE-C LSE-D speaker similarity (WavLM-TDNN) MOS
Baselines: AV-TranSpeech ASR+NMT+TTS+TFG cascade AVSR+NMT+TTS+TFG cascade A2T+TTS+TFG AV2T+TTS+TFG A2A+TFG
Builds on: AV-HuBERT HuBERT Wav2Lip HiFi-GAN Direct speech-to-speech translation with discrete units (Lee et al.)
Closest papers by meaning
- video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.86)
- ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.86)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.85)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.85)
- Visual Hallucination Elevates Speech Recognition (AAAI 2024 · similarity 0.85)
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.