Research map / Vision-language and multimodal / Cross-modal learning
MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoHAVE, a sparse mixture-of-experts AVSR decoder with audio and visual expert groups and a hierarchical (inter-modal plus intra-modal) gating mechanism trained with a load biasing loss, improving noisy LRS3 and MuAViC robustness.
Problem
Existing audio-visual speech recognition (AVSR) models are small (under 0.5B parameters) and struggle to scale capacity without hurting efficiency, and they underuse visual information under audio noise.
Approach
Split decoder MoE experts into audio and visual groups; an inter-modal router softly weights the two groups per token, guided by a group-level load biasing loss that pushes audio-only tokens toward the audio group and video-only tokens toward the visual group, with top-1 intra-modal routing inside each group.
Key results
- MoHAVE-Large (1.0B total, 553M active) reaches 4.5% N-WER on noisy LRS3 versus 5.7% for AV-HuBERT-Large; with CMA encoder it reaches 4.2%.
- MoHAVE-Base (359M total, 189M active) reaches 5.8% N-WER, comparable to AV-HuBERT-Large at 477M, with 14.8 vs 12.1 GFLOPs per sequence against AV-HuBERT-Base.
- On MuAViC multilingual noisy AVSR, MoHAVE-Large achieves 37.4% average WER versus 38.9% for re-implemented mAV-HuBERT, and 19.5% average X-En BLEU versus 18.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hierarchical audio-visual expert routing with load biasing | AV-HuBERT-Base · LRS3 noisy | N-WER: 6.6 -> 5.8 | Helped |
| Add hierarchical audio-visual expert routing with load biasing | AV-HuBERT-Large · LRS3 noisy | N-WER: 5.7 -> 4.5 | Helped |
| Replace standard MoE routing with modality-specific hard routing | AV-MoE-Base, 8 experts · LRS3 noisy | N-WER: 6.3 -> 5.9 | Helped |
| Replace standard MoE routing with modality-specific hard routing | AV-MoE-Large, 8 experts · LRS3 noisy | N-WER: 5.7 -> 4.8 | Helped |
| Add load biasing (ablation) | MoHAVE-Base · LRS3 noisy | N-WER: 6.2 -> 5.8 | Helped |
| Add load biasing (ablation) | MoHAVE-Large · LRS3 noisy | N-WER: 5.6 -> 4.5 | Helped |
| Add a second visual expert while keeping one audio expert (ablation) | MoHAVE-Base · LRS3 noisy | N-WER: 5.8 -> 5.7 | Helped |
| Add a second audio expert while keeping one visual expert (ablation) | MoHAVE-Base · LRS3 noisy | N-WER: 5.8 -> 6.2 | Hurt |
| Activate two audio and two visual experts instead of one from each group (ablation) | MoHAVE-Base · LRS3 noisy | N-WER: 5.8 -> 6.9 | Hurt |
| Add hierarchical audio-visual expert routing | mAV-HuBERT · MuAViC noisy, non-English AVSR | average WER: 38.9 -> 37.4 | Helped |
| Add hierarchical audio-visual expert routing | mAV-HuBERT · MuAViC noisy, X-En AVS2TT | average BLEU: 18.5 -> 19.5 | Helped |
| Use hard routing instead of adaptive hierarchical routing | mAV-HuBERT · MuAViC noisy, non-English AVSR | average WER: 38.9 -> 39.3 | Hurt |
Surprising findings
- Simple MoE (AV-MoE) without modality-aware grouping gives only marginal gains over the dense AV-HuBERT baseline.
- Removing the load biasing loss from MoHAVE drops it to roughly the AV-MoE level, showing the group specialization is what drives gains.
Limitations
- Clean-speech gains are not significant; MoHAVE mainly helps under noise.
- Multilingual results rely on a larger re-implemented pretrained AV-HuBERT, and AV-MoE/EVA comparisons are limited by differing tasks and unreleased code.
Experimental setup
Datasets: LRS3 MUSAN VoxCeleb2 MuAViC DEMAND LRS3-TED mTEDx
Benchmarks: LRS3 MuAViC DEMAND-augmented LRS3
Metrics: WER N-WER C-WER BLEU FLOPs
Baselines: AV-HuBERT AV-MoE AV-MoE with hard routing BRAVEn MIR-GAN UniVPM CMA Whisper large-v2 mAV-HuBERT XLAVS-R 300M XLAVS-R 2B u-HuBERT
Builds on: AV-HuBERT sparsely-gated mixture of experts V/T-MoE hard routing hierarchical mixture of experts
Closest papers by meaning
- MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition (NeurIPS 2025 · similarity 0.88)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.86)
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.86)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.86)
- $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time (NeurIPS 2025 · similarity 0.86)
- MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.