Research map / Architectures and efficiency / Mixture of experts
No Need to Talk: Asynchronous Mixture of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMALLTALK LM, a hard mixture of independently trained language models whose tiny (4.4M) language-model routers pick an expert from a short prefix, enabling near-asynchronous training and cheap inference.
Problem
Training large LMs requires high-bandwidth synchronization across nodes, and sparse/MoE inference requires per-token routing with all parameters in memory; prior asynchronous mixture methods rely on full-corpus clustering or metadata, which is impractical for short prefixes.
Approach
Route sequences with independent small language models (routers) that score a short prefix via likelihood, trained with an EM-style scheme using balanced assignments, so each expert is trained completely independently on its data shard with only a small amount of loss communication.
Key results
- A 335M-parameter mixture with 32 experts reaches test perplexity 9.07, beating the 1.3B dense baseline (9.10) at roughly equal training FLOPs and about one-third the inference FLOPs.
- The 1.3B model with 32 experts gains up to ~18% lower perplexity than the dense baseline with under 1% extra training FLOPs and under 3% extra inference FLOPs.
- On zero-shot downstream tasks, the 32-expert 1.3B mixture beats the dense baseline on four of five benchmarks (gains of 3%, 2%, 3%, 1% on ARC-C, ARC-E, HellaSwag, MMLU) and on 42 of 56 MMLU subtasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a dense model with a 335M SMALLTALK LM mixture using 4 experts | 335M language model · RedPajama-V2 test | perplexity: 11.78 -> 10.78 | Helped |
| Replace a dense model with a 335M SMALLTALK LM mixture using 8 experts | 335M language model · RedPajama-V2 test | perplexity: 11.25 -> 10.20 | Helped |
| Replace a dense model with a 335M SMALLTALK LM mixture using 16 experts | 335M language model · RedPajama-V2 test | perplexity: 10.80 -> 9.64 | Helped |
| Replace a dense model with a 335M SMALLTALK LM mixture using 32 experts | 335M language model · RedPajama-V2 test | perplexity: 10.5 -> 9.07 | Helped |
| Replace a dense model with a 1.3B SMALLTALK LM mixture using 4 experts | 1.3B language model · RedPajama-V2 test | perplexity: 9.10 -> 8.75 | Helped |
| Replace a dense model with a 1.3B SMALLTALK LM mixture using 16 experts | 1.3B language model · RedPajama-V2 test | perplexity: 8.48 -> 7.42 | Helped |
| Replace a dense model with a 1.3B SMALLTALK LM mixture using 32 experts | 1.3B language model · RedPajama-V2 test | perplexity: 8.20 -> 6.76 | Helped |
| Use 32 SMALLTALK LM experts instead of the dense baseline | 1.3B language model · ARC Challenge | accuracy: 0.2892 -> 0.3200 | Helped |
| Use 32 SMALLTALK LM experts instead of the dense baseline | 1.3B language model · ARC Easy | accuracy: 0.6549 -> 0.6780 | Helped |
| Use 32 SMALLTALK LM experts instead of the dense baseline | 1.3B language model · HellaSwag | accuracy: 0.4663 -> 0.4912 | Helped |
| Use 32 SMALLTALK LM experts instead of the dense baseline | 1.3B language model · SciQ | accuracy: 0.9130 -> 0.8950 | Hurt |
| Use 32 SMALLTALK LM experts instead of the dense baseline | 1.3B language model · MMLU abstract algebra | accuracy: 0.2300 -> 0.1400 | Hurt |
Surprising findings
- Router size (4.4M to 335M parameters) has practically no effect on mixture perplexity, so tiny routers suffice.
- Routing with only 32 prefix tokens still outperforms the dense baseline.
Limitations
- Experiments are limited to models up to 1.3B parameters and up to 32 experts.
- Routing uses a prefix of 256 tokens in the main setting, which may be too long for some applications like conversational AI.
- Downstream evaluation is zero-shot and the dense baseline wins on several tasks (e.g., SciQ, many MMLU subtasks).
Experimental setup
Datasets: RedPajama-V2 ARC Challenge ARC Easy HellaSwag SciQ MMLU
Benchmarks: ARC Challenge ARC Easy HellaSwag SciQ MMLU
Metrics: test perplexity zero-shot accuracy training FLOPs inference FLOPs
Baselines: Dense transformer LM (335M and 1.3B) TF-IDF + balanced K-Means routing Switch Transformer-style token routing (discussed)
Builds on: Switch Transformer Expectation-Maximization mixture training K-Means clustering of data (TF-IDF/SVD)
Closest papers by meaning
- Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.89)
- Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.89)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.88)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.88)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.88)
- EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.