Research map / Architectures and efficiency / Mixture of experts
Layerwise Recurrent Router for Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RMoE, which adds a shared GRU across layers to produce layer-wise recurrent routing states that condition each layer's MoE router, improving perplexity/BPC and expert diversity over SMoE and router variants.
Problem
Routers in different MoE layers act independently and do not share routing information, which may lead to suboptimal token-expert assignments and parameter inefficiency in MoE language models.
Approach
Treat routing decisions across consecutive layers as a sequence and pass a recurrent GRU state across layers (with per-layer input projectors) so each layer's router is conditioned on previous layers' routing; this gives cross-layer information sharing and an extra recurrent gradient path to the router, with negligible overhead.
Key results
- On Enwiki8 medium setting, RMoE reaches 1.116 test BPC vs 1.128 for SMoE and 33.06 to 32.87 test PPL on WikiText-103 vs SMoE.
- With 0.91B-parameter models pretrained on 40B tokens, RMoE averages 55.31 on lm-eval tasks vs 54.26 for SMoE, and 57.15 vs 56.13 after SFT.
- Plugging the GRU router into XMoE improves it at all lower dimensions (8, 16, 32), e.g. XMoE (16) test PPL 32.93 to 32.47.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add layerwise recurrent GRU routing | SMoE · Enwiki8 test | BPC: 1.128 -> 1.116 | Helped |
| add layerwise recurrent GRU routing | SMoE · WikiText-103 test | PPL: 33.061 -> 32.867 | Helped |
| add layerwise recurrent GRU routing | SMoE · GSM8K, 120B-token pretraining | accuracy: 34.8 -> 36.13 | Helped |
| add layerwise recurrent GRU routing | SMoE · MMLU, 120B-token pretraining | accuracy: 52.79 -> 54.02 | Helped |
| add layerwise recurrent GRU routing | SMoE · GSM8K, 400B-token pretraining | accuracy: 52.16 -> 52.99 | Helped |
| add layerwise recurrent GRU routing | SMoE · MMLU, 400B-token pretraining | accuracy: 59.54 -> 60.60 | Helped |
| add a larger MLP router instead of a linear router (ablation) | SMoE · Enwiki8 test | BPC: 1.128 -> 1.137 | Hurt |
| add a larger MLP router instead of a linear router (ablation) | SMoE · WikiText-103 test | PPL: 33.061 -> 33.142 | Hurt |
| remove layerwise recurrent-state passing (ablation) | RMoE · Enwiki8 test | BPC: 1.116 -> 1.123 | Hurt |
| preserve gradient flow through the previous recurrent state (ablation) | RMoE · Enwiki8 test | BPC: 1.133 -> 1.116 | Helped |
| add previous-layer routing-logit residuals without recurrence (ablation) | RMoE-NP · Enwiki8 test | BPC: 1.123 -> 1.124 | No clear effect |
| use layerwise projectors instead of one shared projector (ablation) | GRU router · Enwiki8 test | BPC: 1.123 -> 1.116 | Helped |
Surprising findings
- Removing the cross-layer recurrent state (RMoE+NP, stateless GRU with same parameters) drops performance below SMoE on Enwiki8 (test BPC 1.123 vs. 1.116 for RMoE, and worse than SMoE in the small setting).
- Detaching the recurrent hidden state across layers hurts performance more than removing cross-layer state entirely, indicating the recurrent gradient path matters.
Limitations
- Gains on large-scale pretraining are modest (around one point on average) and the 0.91B experiments are under-trained, with near-random results on hard tasks like MMLU.
- Speedup and memory are reported only in limited settings; the GRU adds about 0.47M parameters in small models and the recurrence is sequential across layers.
Experimental setup
Datasets: enwiki8 WikiText-103 Alpaca multilingual pretraining corpus (Wikipedia, finance, legal)
Benchmarks: enwiki8 WikiText-103 ARC-Easy HellaSwag PIQA SciQ LAMBADA MMLU GSM8K HumanEval
Metrics: BPC perplexity accuracy acc-norm training speed peak memory mutual information gate entropy expert similarity
Baselines: SMoE HyperMoE SMoE-MLP RandomMoE XMoE CosineSMoE
Builds on: Switch Transformer XMoE HyperMoE SMoE-Dropout CompeteSMoE DeepSeekMoE
Closest papers by meaning
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.91)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.90)
- Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts (NeurIPS 2026 · similarity 0.90)
- Ada-K Routing: Boosting the Efficiency of MoE-based LLMs (ICLR 2025 · similarity 0.90)
- ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.