Research map / Architectures and efficiency / Mixture of experts
MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoME, which inserts sparse top-k routed and shared experts with a shared router into a frozen Llama 3 LLM trained over multiple audio-visual token granularities, for elastic-rate AVSR, ASR and VSR.
Problem
LLM-based audio-visual speech recognition is costly due to long token sequences, and existing Matryoshka representation learning approaches fix monolithic per-scale representations, trained independently per scale, which hurts high-compression performance.
Approach
Integrates sparse Mixture-of-Experts into Matryoshka-trained LLMs as lightweight bottleneck adapters placed parallel to MHSA/FFN/layer in a frozen LLM, with a router and experts shared across all audio-visual token scales so that expert activations align across granularities, plus shared experts for scale-invariant knowledge.
Key results
- On LRS3 at (4,2) compression, MoME-23/4-LAYER reaches 1.5% WER with 3.5M active parameters versus 2.4% for Llama-AVSR (27.5M active parameters, separately trained per rate).
- A single MoME-23/4-LAYER with bottleneck dimension 1 uses 0.9M active MoME parameters on LRS3 and stays close to the best variants (2.0% WER at (4,2)).
- Under babble noise on LRS2 at 7.5 dB SNR, MoME-23/4-LAYER gets 4.8% WER versus 5.6% for Llama-AVSR and 6.2% for Llama-MTSK MS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Insert MoME parallel to the full LLM layer rather than use Llama-MTSK SS | Llama 3.2-3B AVSR · LRS2 (4,2) | WER (%): 3.4 -> 2.7 | Helped |
| Insert MoME parallel to the full LLM layer rather than use Llama-MTSK SS | Llama 3.2-1B AVSR · LRS3 (4,2) | WER (%): 2.3 -> 1.5 | Helped |
| Use MoME-23/4-LAYER instead of Llama-AVSR | AVSR · LRS2, LRS3; all four listed compression settings | WER (%) | Helped |
| Add one shared expert to the 4-routed-expert model (ablation) | MoME-MHSA · LRS2 AVSR (4,2) | WER (%): 3.3 -> 3.2 | Helped |
| Add a second shared expert to the 23-routed-expert model (ablation) | MoME-MHSA · LRS2 AVSR | WER (%): 2.9 -> 2.8 (4,2); 4.3 -> 4.7 (16,5) | Mixed |
| Increase the number of routed experts from 1 to 23 while adjusting expert bottleneck size (ablation) | MoME-MHSA · LRS2 AVSR (4,2) | WER (%): 3.4 -> 2.9 | Helped |
| Increase active experts from 1 to 8 (ablation) | MoME-8/k-MHSA · LRS2 AVSR (4,2) | WER (%): 3.3 -> 3.1 | Helped |
| Use separate granularity-specific routers instead of a shared router (ablation) | MoME-15/3-LAYER · LRS2 AVSR (16,2) | WER (%): 4.0 -> 4.7 | Hurt |
| Use separate granularity-specific routers instead of a shared router (ablation) | MoME-15/3-LAYER · LRS3 AVSR (16,5) | WER (%): 3.4 -> 3.7 | Hurt |
| Increase loss weights for lower scales to [1,1,1.5,2] (ablation) | MoME-15/3-LAYER · LRS2 AVSR (4,2) | WER (%): 2.9 -> 4.0 | Hurt |
| Increase loss weights for lower scales to [1,1,1.5,2] (ablation) | MoME-15/3-LAYER · LRS3 AVSR (4,5) | WER (%): 1.9 -> 2.2 | Hurt |
| Increase compression from (1,1) to (16,5) | MoME-15/3-MHSA · 23-second speech input | inference time (s): 12.75 -> 6.74 | Helped |
Surprising findings
- Using a separate router per granularity (disjoint routers) degrades WER, especially at high compression, supporting the value of the shared router.
- Increasing the number of shared experts beyond one does not consistently help and can hurt at high compression (e.g., (16,5)).
Limitations
- All experts must reside in memory, so MoME has higher memory footprint than dense adapters despite lightweight experts.
- Inference cost is slightly higher than Llama-AVSR and Llama-MTSK due to router and expert dispatch overhead.
- Evaluated only on audio-visual speech tasks; applicability to other multimodal domains is untested.
Experimental setup
Datasets: LRS2 LRS3 NOISEX
Benchmarks: LRS2 LRS3
Metrics: WER
Baselines: Llama-AVSR Llama-MTSK MS Llama-MTSK SS Llama-MTSK MSS MMS-Llama UniVPM USR Whisper-Flamingo Auto-AVSR AV-HuBERT
Builds on: Matryoshka Representation Learning Llama-MTSK Llama-AVSR DeepSeekMoE Switch-style load balancing loss
Closest papers by meaning
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.88)
- MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition (ICML 2025 · similarity 0.88)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.87)
- Matryoshka Query Transformer for Large Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.86)
- video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.