Research map / Architectures and efficiency / Mixture of experts
$\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Refactors a pre-trained dense LLM (Llama-2-7B-chat) into a smaller MoE via activation-sparsity expert pruning and a decoupled autoregressive pre-gating router, then co-designs expert-aware batching, prefetching, and Belady-inspired caching for inference.
Problem
Serving MoE LLMs is inefficient because layer-wise routers prevent prefetching and expert-aware batching, and training MoEs from scratch is costly; the paper targets converting a pre-trained dense LLM into a smaller MoE with system-friendly inference.
Approach
Observes that layer-wise router expert selections are highly correlated across adjacent layers, so replaces them with a single causal-attention pre-gating router decoupled from the MoE backbone; this lets all routing paths be known before inference, enabling expert-aware batching, fine-grained prefetching and Belady-style caching.
Key results
- On downstream tasks the refactored 4.7B-activated MoE (from Llama-2-7B, 1B tokens) reaches 55.5% average, above Sheared-Llama (53.2%) and Pythia-2.8B (48.3%), with MMLU 38.9%.
- Expert-aware batching reduces mean latency by 5.0-6.1% and p95 latency by 9.5-10.0% versus decoding- or prefill-prioritized batching, lowering unique experts per batch from about 5.1 to 3.51.
- Temporal locality: 2921 of 4096 tokens follow the last token's expert choice versus 850 in Mixtral-8x7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| refactorize a 4.7B dense model into an MoE (ablation) | Llama-2-7B-chat · MMLU | accuracy: 27.1% -> 38.9% | Helped |
| refactorize a 4.7B dense model into an MoE (ablation) | Llama-2-7B-chat · Arxiv, Books, C4, Common Crawl, Github, StackExchange, Wikipedia | perplexity: 5.63 -> 4.18; 1.94 -> 1.31; 11.78 -> 10.57; 9.68 -> 7.72; 3.75 -> 2.39; 13.42 -> 12.52; 6.24 -> 3.94 | Helped |
| convert Mistral without prior knowledge of its pretraining domain (ablation) | Read-ME · MMLU | accuracy: 62.1% -> 39.2% | Hurt |
| convert Mistral without prior knowledge of its pretraining domain (ablation) | Read-ME · Downstream task evaluation | average accuracy: 69.4% -> 60.0% | Hurt |
| use expert-aware batching based on pre-gated expert requirements instead of decoding- or prefill-prioritized batching | Read-ME inference system · Batched inference (Chatbot Arena workload) | mean latency: 5.0-6.1% | Helped |
| use expert-aware batching based on pre-gated expert requirements instead of decoding- or prefill-prioritized batching | Read-ME inference system · Batched inference (Chatbot Arena workload) | p95 latency: 9.5-10.0% | Helped |
| enable exact prefetching rather than on-demand loading | Read-ME inference system · Inference with varying expert-cache capacities | end-to-end latency: up to 30% | Helped |
| refactor the dense MLP into MoE | Read-ME · Single isolated inference request | latency: 19% | Helped |
| use a shared autoregressive router rather than conventional layerwise routers | Read-ME inference system · Single isolated inference request | router latency contribution: 3.95% -> 0.4% | Helped |
| use autoregressive pre-gating rather than Mixtral-8×7B routing | Read-ME router · 4096-token sequence | tokens following the previous token's expert choice: 850 -> 2921 | Helped |
Surprising findings
- Mixtral's layer-wise router decisions are nearly deterministic given the previous layer's pair of experts, with over 70% of tokens following a single transition.
- Converting to a 4.7B dense model of matching size yields only 27.1% MMLU versus 38.9% for the MoE refactorization with the same data.
Limitations
- Serving system is designed for a single accelerator; distributed serving is left to future work.
- Temporal locality from the auto-regressive router only holds within a single request, not across requests.
Experimental setup
Datasets: RedPajama Chatbot Arena Conversation Dataset Arxiv C4 Wikipedia
Benchmarks: MMLU HellaSwag WinoGrande ARC-Easy ARC-Challenge LogiQA CoQA
Metrics: accuracy normalized latency p95 latency cache hit ratio perplexity
Baselines: Sheared-Llama Pythia Open-Llama-v2 OpenMoE LLM-Pruner SliceGPT LaCo Compresso Decoding-prioritized batching Prefill-prioritized batching LRU caching
Builds on: Mixtral Pre-gated MoE SiDA-MoE Sheared-Llama Belady optimal caching
Closest papers by meaning
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.92)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.91)
- Ada-K Routing: Boosting the Efficiency of MoE-based LLMs (ICLR 2025 · similarity 0.91)
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.90)
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.90)
- BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.