Research map / Vision-language and multimodal / Cross-modal learning
MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
Yuxuan Lou, Kai Yang, Yang You
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use modality-aware routing instead of vanilla MoE routing (ablation) | MoST · ASR/TTS (LibriSpeech dev-clean), sWUGGY, LlamaQ at 10k steps | WER / accuracy | Improved |
| add shared experts to modality-specific experts (ablation) | MoST · ASR/TTS (LibriSpeech dev-clean), sWUGGY, LlamaQ at 10k steps | WER / accuracy | Improved |
| MoST-style upcycling instead of traditional MoE upcycling (Llama3.2 3B init) (ablation) | Llama3.2 3B · controlled comparison, speech-text tasks | ASR improvement: 7.3% improvement in ASR to 21.8% in SQA | Improved |
| use MoE upcycling instead of a dense baseline from identical init (ablation) | Llama3.2 3B · controlled comparison, speech-text tasks | task scores | Improved |
| modality-aware routing instead of vanilla MoE routing (ablation) | MoST · routing analysis during training | routing entropy and Gini coefficient | Improved |
| stronger initialization (DeepSeek-V2 Lite) | MoST · absolute performance | task scores | Improved |
| include shared experts to mitigate forgetting of text abilities | MoST · text benchmarks (MMLU, TriviaQA, GSM8K, HumanEval) | accuracy: MMLU 55.4 vs MinMo 58.5; GSM8K 66.9; HumanEval 58.2 | Mixed |
Closest papers by meaning
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.89)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
- Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)
- mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
