Research map / Large language models / Instruction and fine-tuning

$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts

ICML 2024 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MoE-RBench, a reliability benchmark assessing sparse MoE LLMs versus dense counterparts on safety/hallucination, adversarial robustness, and OOD robustness, and studies training and inference recipes to improve MoE reliability.

Problem

Reliability of sparse Mixture-of-Experts language models is under-evaluated compared with dense LLMs, and MoE models sometimes underperform dense models when fine-tuned on new domains.

Approach

A multi-dimensional reliability benchmark for SMoE that compares MoE models against matched dense models on harmfulness, hallucination, adversarial NLI, and style-based OOD robustness, plus an empirical study of MoE-specific training settings (aux loss, expert dropout, router training) and contrastive decoding (DoLa).

Key results

• Switch-base outperforms t5-base by about 2.1% on adversarial RA and 2.2% on standard RA; MoLM-350M-K2 beats pythia-410M by 2.6% on adversarial RA.
• On TruthfulQA MC, dense models outperform all MoLM variants and LlamaMoE-3.5B-K2, while MoE models win on Natural Questions exact match (e.g., LlamaMoE-3.5B-K4 19.92 vs OpenLlama-3B 16.09).
• Non-zero load-balancing loss (1e-3) raises generalization for switch-base (50.41 vs 49.63) and MoLM-350M-K2 (47.31 vs 45.16); larger expert dropout (0.2-0.3) improves accuracy and generalization.

Measured effects reported in the paper

Surprising findings

• Bi-level (alternating router/backbone) training gives minimal or negative gains for LLM MoE, unlike CNN MoE results.
• Larger MoLM variant underperforms smaller one on TruthfulQA, an inverse-scaling pattern extended to MoE.

Limitations

• Relies on small models (up to about 3.5B) and few MoE architectures, with mismatched dense counterparts in size and training data.
• Adversarial and OOD evaluations use NLI and sentiment tasks; safety evaluation relies on automated judges rather than human annotation.

Experimental setup

Datasets: Alpaca SNLI ANLI SST-2 BOSS MaliciousInstructions CoNa Controversial Do-not-answer TruthfulQA Natural Questions

Benchmarks: SNLI SNLI-hard ANLI SST-2 style OOD BOSS TruthfulQA Natural Questions MaliciousInstructions CoNa Controversial Do-not-answer

Metrics: accuracy harmfulness score unsafe ratio exact match MC1/MC2/MC3

Baselines: T5-base pythia-410M pythia-1.4B OpenLlama-3B

Builds on: Switch Transformers ModuleFormer LLaMA-MoE DoLa BOSS benchmark

Closest papers by meaning

• Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
• Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.89)
• Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · similarity 0.89)
• Mixture of Experts Made Intrinsically Interpretable (ICML 2025 · similarity 0.89)
• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.89)