Research map / Architectures and efficiency / Mixture of experts
Mixture of Lookup Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture of Lookup Experts (MoLE), which trains MoE-like experts on embedding-layer outputs and re-parameterizes them into offloadable lookup tables so inference needs no expert computation or per-step expert loading.
Problem
MoE models must keep all experts in VRAM, and expert offloading adds large per-step parameter transfer latency, especially with batched generation.
Approach
Feed experts with the embedding-layer output (a finite set determined by input token id) and activate all experts during training; after training, precompute each expert's output for every vocabulary token into LUTs that are offloaded and only a single row per token is loaded at inference.
Key results
- MoLE-16E (160M activated) averages 41.9 vs 38.8 for dense and 40.3 for MoE-10E on eight zero-shot benchmarks, with about 0.15M parameters loaded per token versus 57M for MoE-34E.
- At 410M activated parameters, MoLE-4E averages 44.6 vs 43.9 for MoE-10E and 41.8 for dense.
- Decoding latency of MoLE stays close to the dense model and does not grow with batch size, while MoE with expert offloading gets slower as batch size increases (per the efficiency figure).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoLE-4E instead of Dense | 160M activated-parameter models · Zero-shot benchmarks | AVG accuracy: 38.8 -> 40.8 | Helped |
| Use MoLE-4E instead of MoE-10E | 160M activated-parameter models · Zero-shot benchmarks | AVG accuracy: 40.3 -> 40.8 | Helped |
| Use MoLE-16E instead of MoE-34E | 160M activated-parameter models · Zero-shot benchmarks | AVG accuracy: 41.8 -> 41.9 | Helped |
| Use MoLE-4E instead of Dense | 410M activated-parameter models · Zero-shot benchmarks | AVG accuracy: 41.8 -> 44.6 | Helped |
| Use MoLE-4E instead of MoE-10E | 410M activated-parameter models · Zero-shot benchmarks | AVG accuracy: 43.9 -> 44.6 | Helped |
| Use MoLE-16E instead of MoE-34E | 410M activated-parameter models · Zero-shot benchmarks | AVG accuracy: 46.6 -> 45.7 | Hurt |
| Use MoLE-4E instead of Dense or MoE-10E | 1B activated-parameter models · Zero-shot benchmarks | AVG accuracy: 44.3 -> 47.4; 46.6 -> 47.4 | Helped |
| Add load balance loss to MoLE-16E training (ablation) | MoLE-16E, 160M activated parameters · Zero-shot benchmarks | AVG accuracy: 41.9 -> 41.7 | Hurt |
| Add load balance loss and z-loss to MoLE-16E training (ablation) | MoLE-16E, 160M activated parameters · Zero-shot benchmarks | AVG accuracy: 41.9 -> 40.6 | Hurt |
| Set routed expert hidden dimension to 768 instead of 3072 (ablation) | MoLE-16E, 160M activated parameters · Zero-shot benchmarks | AVG accuracy: 41.9 -> 40.8 | Hurt |
| Set routed expert hidden dimension to 12288 instead of 3072 (ablation) | MoLE-16E, 160M activated parameters · Zero-shot benchmarks | AVG accuracy: 41.9 -> 41.7 | No clear effect |
| Increase routed experts from 2 to 16 (ablation) | MoLE, 160M activated parameters · Zero-shot benchmarks | AVG accuracy: 39.7 -> 41.9 | Helped |
Surprising findings
- Adding standard MoE auxiliary losses (load balance and z-loss) degraded MoLE performance.
- Fully activating MoE experts without a router (full activation) improved over top-2 MoE by about 1.5 points on average.
Limitations
- LUT storage is 2.4 to 7.4 times larger than the offloaded MoE experts, and grows with vocabulary size and number of experts.
- Experts receive only token embeddings without context, which costs about 0.7 points on average versus contextual inputs in the ablation.
- Evaluated only at small scales (160M to 1B activated parameters) on a 100B-token Pile subset.
Experimental setup
Datasets: deduped Pile (100B-token subset)
Benchmarks: ARC-C ARC-E BoolQ HellaSwag PIQA RACE SIQA LAMBADA
Metrics: zero-shot accuracy decoding latency parameters loaded per token
Baselines: Dense model MoE-10E MoE-34E MoE with expert offloading
Builds on: Mixture of Experts Pythia Mixtral ZeRO-Infinity
Closest papers by meaning
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.91)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.90)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.89)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.89)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.89)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.