Research map / Architectures and efficiency / Mixture of experts
SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimulMEGA, an unsupervised read/write policy learning framework for simultaneous speech translation and streaming TTS that uses a Mixture-of-Experts refiner with a sigmoid routing gate, adding no inference overhead.
Problem
Simultaneous speech translation must balance translation quality, latency, and coherence, and multilingual many-to-many settings with divergent read/write policies make unified policy learning hard; streaming TTS also lacks fine-grained low-latency solutions.
Approach
Train an auxiliary MoE refiner whose two experts (prefix cross-attention expert and global pooled-embedding expert) are mixed by a sigmoid gate; the gate's score implicitly defines a read/write policy with threshold inference, with a previous-output attention to prevent global leakage and a normalization loss to control score distribution.
Key results
- SimulMEGA-S2T (561M) shows under 7% BLEU degradation at 1.5 s AL and under 3% at 3 s AL, versus about 12% for Seamless at 2 s AL.
- On CoVoST2 X-EN, SimulMEGA-S2T reaches 36.9 BLEU vs 37.0 offline base, while Seamless-S2T (2.0B) reaches 35.3.
- SimulMEGA-TTS achieves WER 2.54 and SIM 0.661 on LibriSpeech with AL 1.2 vs CosyVoice2-S-ZS AL 18.5 and WER 5.31.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SimulMEGA-S2T instead of Seamless-S2T | Simultaneous S2TT · CoVoST2 X-EN | BLEU: 35.3 -> 36.9 | Helped |
| Use SimulMEGA-S2T instead of Seamless-S2T | Simultaneous S2TT · Fleurs X-EN | BLEU: 28.0 -> 26.3 | Hurt |
| Add pre-Sigmoid Gaussian noise with sigma=1 (ablation) | SimulMEGA router · CoVoST2 EN-ZH | BLEU | Mixed |
| Add router score normalization (ablation) | SimulMEGA router · CoVoST2 EN-ZH | latency-performance curve | Helped |
| Keep the offline training loss during simultaneous training (ablation) | SimulMEGA · CoVoST2 EN-ZH | BLEU: approximately 1 BLEU point | Helped |
| Keep the prefix training loss during simultaneous training (ablation) | SimulMEGA · CoVoST2 EN-ZH | BLEU | No clear effect |
Surprising findings
- Removing the prefix loss L_prefix causes negligible degradation, while removing the offline loss reduces BLEU by about 1 point.
- DiG-SST and AlignATT perform poorly on all X-to-ZH pairs, which the authors attribute to the UTF-8 byte-level BPE tokenizer splitting CJK characters into multiple tokens.
Limitations
- Streaming TTS supports only two languages (Chinese and English), and the S2ST system is cascaded with inconsistent text tokenizers and about 0.1 s extra latency.
- Input is limited to 30 seconds per segment, so a VAD model is still needed to segment streams.
- Training data relies on pseudo translation labels from a cloud translation API rather than human-verified translations.
Experimental setup
Datasets: LibriSpeech Multilingual LibriSpeech VoxPopuli Common Voice WenetSpeech KeSpeech Emilia CoVoST2 FLEURS
Benchmarks: CoVoST2 FLEURS LibriSpeech test-clean Seed-TTS test-zh
Metrics: BLEU average lagging (AL) LAAL WER speaker similarity (SIM) ASR-BLEU
Baselines: SeamlessM4T Large-v2 SeamlessStreaming (Seamless-S2T) Wait-K DiG-SST EDATT AlignATT CosyVoice2-S StreamSpeech NAST-S2S
Builds on: Whisper CosyVoice 2 Wait-k policy Monotonic attention DiG-SST
Closest papers by meaning
- Divergence-Guided Simultaneous Speech Translation (AAAI 2024 · similarity 0.91)
- Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation (AAAI 2025 · similarity 0.88)
- High-Fidelity Simultaneous Speech-To-Speech Translation (ICML 2025 · similarity 0.87)
- SimulPL: Aligning Human Preferences in Simultaneous Machine Translation (ICLR 2025 · similarity 0.86)
- Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity (ICLR 2025 · similarity 0.85)
- Objective Soups: Multilingual Multi-Task Modeling for Speech Processing (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.