Research map / Architectures and efficiency / Mixture of experts
Mixture of Experts Made Intrinsically Interpretable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoE-X, a mixture-of-experts LM that is intrinsically interpretable by rewriting MoE as a wide sparse MLP, using ReLU experts and sparsity-aware routing, evaluated on chess PGN modeling and FineWeb language modeling.
Problem
Neurons in LLM MLPs are polysemantic and hard to interpret; post-hoc methods like SAEs are costly and incomplete, and prior intrinsically interpretable architectures either work only at toy scale or hurt performance.
Approach
View a sparse MoE layer as a wide, structurally sparse MLP, enforce activation sparsity inside experts with ReLU, and route tokens to the experts whose pre-activations are predicted to be sparsest using a Gaussian-based erf estimate of the l0 norm.
Key results
- On chess, MoE-X reaches 0.428 coverage and 0.840 reconstruction versus 0.356 and 0.608 for dense GPT-2, and 0.424 and 0.734 for Switch.
- On FineWeb-trained language models, MoE-X-M (8x354M) achieves 14.78 OpenWeb PPL, beating GPT-2 Medium at 17.19, while Switch-S is best on OpenWeb and LAMBADA.
- Sparsity-aware gating scores show a strong negative correlation (r<-0.95) with actual expert l0 sparsity, and detection-score interpretability of MoE-X Medium surpasses GPT-2 with SAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoE-X instead of Switch | Chess model · Chess validation | Coverage: 0.424 -> 0.428 | Helped |
| Use MoE-X instead of Switch | Chess model · Chess validation | Reconstruction: 0.734 -> 0.840 | Helped |
| Use MoE-X instead of Switch | Small language model · OpenWeb | perplexity: 18.36 -> 19.42 | Hurt |
| Use MoE-X instead of Switch | Small language model · WikiText103 | perplexity: 45.22 -> 43.80 | Helped |
| Use MoE-X instead of Switch | Medium language model · OpenWeb | perplexity: 15.43 -> 14.78 | Helped |
| Use MoE-X instead of Switch | Medium language model · LAMBADA | perplexity: 20.82 -> 21.34 | Hurt |
| Add ReLU experts (ablation) | MoE-X with Sparsity Router · Chess validation | Coverage: 0.404 -> 0.428 | Helped |
| Add ReLU experts (ablation) | MoE-X with Sparsity Router · Chess validation | Reconstruction: 0.740 -> 0.840 | Helped |
| Add Sparsity Router (ablation) | MoE-X with ReLU experts · Chess validation | Coverage: 0.418 -> 0.428 | Helped |
| Add Sparsity Router (ablation) | MoE-X with ReLU experts · Chess validation | Reconstruction: 0.829 -> 0.840 | Helped |
| Upcycle dense model weights rather than train MoE-X from scratch (ablation) | MoE-X · Chess BSP interpretability | Coverage: 0.398 -> 0.428 | Helped |
| Upcycle dense model weights rather than train MoE-X from scratch (ablation) | MoE-X · Chess BSP interpretability | Reconstruction: 0.657 -> 0.840 | Helped |
Surprising findings
- Post-hoc SAE on GPT-2 substantially degrades language modeling performance (OpenWeb PPL 22.83 to 31.60).
- SoLU, a design claimed to improve interpretability, scores below the GELU GPT-2 baseline on coverage and reconstruction.
Limitations
- Interpretability is measured mainly on chess board-state properties, a proxy with a known ground truth, and on auto-interp detection scores for language.
- Language gains are mixed: Switch Transformer beats MoE-X on OpenWeb and LAMBADA perplexity at the small scale.
- Each expert is under-trained relative to a dense model under the same iteration budget, so the comparison needs upcycling to be fair.
Experimental setup
Datasets: lichess_6gb FineWeb 10BT RedPajama-v2 OpenWebText LAMBADA WikiText103 WikiText2
Benchmarks: OpenWebText LAMBADA WikiText103 WikiText2 lichess chess validation
Metrics: validation loss perplexity BSP coverage BSP reconstruction detection accuracy
Baselines: GPT-2 GELU ReLU GEGLU SoLU Monet-HD Monet-VD PEER Switch Transformer GPT-2 with SAE
Builds on: Switch Transformer sparse autoencoders Monet PEER sparse upcycling
Closest papers by meaning
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.91)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.89)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- $\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.