Research map / Large language models / Language, safety and interpretability
Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations
Jeong Hun Yeo, Minsu Kim, Chae Won Kim, Stavros Petridis, Yong Man Ro
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MARC audio-visual data (same 8 languages) beyond MuAViC (ablation) | Cascaded Zero-AVSR · MuAViC avg CER, unseen Russian | avg CER (%): 48.3 -> 28.3 | Improved |
| increase training languages from 8 to 40 in MARC (ablation) | Cascaded Zero-AVSR · Unseen Russian | zero-shot CER (%): 61.0 -> 49.5 | Improved |
| increase training languages from 40 to 81 (ablation) | Cascaded Zero-AVSR · Unseen Russian | zero-shot CER (%): 49.5 -> 40.0 | Improved |
| increase training languages from 8 to 81 (ablation) | Cascaded Zero-AVSR · Unseen Spanish | zero-shot CER (%): 29.9 -> 19.0 | Improved |
| use larger/better de-romanizer LLM (Llama3.2-3B to GPT-4o-mini) (ablation) | Cascaded Zero-AVSR · MuAViC avg of 8 languages | avg CER (%): 35.7 -> 19.5 | Improved |
| use Llama3.1-70B instead of Llama3.1-8B as de-romanizer (ablation) | Cascaded Zero-AVSR · MuAViC avg of 8 languages | avg CER (%): 27.3 -> 21.3 | Improved |
| add Spanish data (same Romance family) to training (ablation) | Cascaded Zero-AVSR · Unseen Spanish | CER (%): 40.1 -> 11.5 | Improved |
| add Turkish and Korean (different family) data (ablation) | Cascaded Zero-AVSR · Unseen Romance languages, French | CER (%): 55.8 -> 59.7 | Hurt |
| add Spanish then Italian data on top of baseline (ablation) | Cascaded Zero-AVSR · Unseen Portuguese | CER (%): 48.0 -> 41.9 | Improved |
| add Spanish data (ablation) | Cascaded Zero-AVSR · Unseen French | CER (%): 55.8 -> 56.6 | No clear effect |
| fine-tune LLM with multi-task training and feed speech features directly (Zero-AVSR vs Cascaded) with same Llama3.2-3B (ablation) | Zero-AVSR · Unseen Russian | zero-shot CER (%): 79.8 -> 45.4 | Improved |
| use GPT-4o-mini for both romanizing and de-romanizing instead of Llama3.1-70B for both (ablation) | Romanization pipeline · MuAViC reconstruction test | avg CER: 13.8 -> 4.0 | Improved |
Closest papers by meaning
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.88)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.88)
- Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation (ICLR 2025 · similarity 0.86)
- MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition (NeurIPS 2025 · similarity 0.86)
- LAMA-UT: Language Agnostic Multilingual ASR Through Orthography Unification and Language-Specific Transliteration (AAAI 2025 · similarity 0.86)
- AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
