Research map / Architectures and efficiency / Mixture of experts

Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference

ICML 2025 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Oracle-MoE, which routes tokens by clustering attention-derived semantic group embeddings (an "oracle space") so consecutive tokens activate the same experts, reducing expert swapping latency for memory-constrained on-device LLM inference.

Problem

MoE-based LLMs on memory-constrained edge devices suffer large latencies because token-level routing causes frequent expert swapping across consecutive tokens due to inconsistent expert activations.

Approach

Route tokens by their semantic group embeddings (averages of token embeddings over attention-defined semantic groups) clustered in a compact oracle space, rather than by individual token embeddings, so that consecutive tokens share experts and swapping drops.

Key results

• On an 8*16 (729M) model with only 1 expert per layer resident, Oracle-MoE adds only about 3s latency versus full-size inference, while Switch Transformer baselines increase latency by up to 2000%.
• Expert swapping accounts for about 50% of Oracle-MoE latency under the tightest memory budget versus over 99% for Switch Transformer.
• Zero-shot downstream performance is comparable to Switch Transformer; at 2.06B, average score is 44.09 vs 43.73.

Measured effects reported in the paper

Surprising findings

• Oracle-MoE's downstream performance matches or slightly exceeds the token-level Switch Transformer despite routing at the semantic-group level.

Limitations

• Evaluated only on GPT-2-based models pretrained on OpenWebText, with sizes up to 2.06B.
• Relies on a token-level MoE warm-up phase and one-time clustering before oracle-space routing.
• Expert prediction results (85-95% accuracy) are reported without explanation of underlying reasons.

Experimental setup

Datasets: OpenWebText TriviaQA GLUE MAG Sci-Cite XSum

Benchmarks: TriviaQA GLUE MAG Sci-Cite XSum

Metrics: expert activation memory-latency curve first token latency F1 accuracy Rouge-1

Baselines: Switch Transformer SwapMoE FIFO swapping LRU swapping

Builds on: Switch Transformer SwapMoE EdgeMoE K-means

Closest papers by meaning

• $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.92)
• Ada-K Routing: Boosting the Efficiency of MoE-based LLMs (ICLR 2025 · similarity 0.90)
• OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.90)
• BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference (AAAI 2025 · similarity 0.89)
• Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.89)
• Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.89)