Research map / Architectures and efficiency / Mixture of experts
Unveiling Super Experts in Mixture-of-Experts Large Language Models
Zunhai Su, Qingyuan Li, HaoZhang, Weihao Ye, Qibo Xue, Yulei Qian, Ngai Wong, Kehong Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune the Super Experts (0.05-0.39% of experts) (ablation) | Qwen3-30B-A3B, DeepSeek-V2-Lite, Mixtral-8x7B · 9 non-reasoning benchmarks | average accuracy: 70.22 -> 55.00 (Qwen3-30B-A3B); 60.27 -> 43.90 (DeepSeek-V2-Lite); 67.84 -> 49.38 (Mixtral) | Hurt |
| prune random experts of equal number (global or layer-controlled) (ablation) | Qwen3-30B-A3B, DeepSeek-V2-Lite, Mixtral-8x7B · 9 non-reasoning benchmarks | average accuracy: 70.22 -> 70.36 (random), 70.21 (LC random) on Qwen3-30B-A3B | No clear effect |
| prune the SEs (ablation) | Qwen3-30B-A3B · GSM8K | accuracy: 89.61 -> 42.38 | Hurt |
| prune the SEs (ablation) | DeepSeek-R1 · reasoning benchmarks (GPQA, Math-500, AIME, LiveCodeBench) | average Pass@1: 75.64 -> 1.81 | Hurt |
| prune the SEs (ablation) | Qwen3-30B-A3B (thinking) · reasoning benchmarks (GPQA, Math-500, AIME, HumanEval) | average Pass@1: 69.37 -> 4.02 | Hurt |
| prune the SEs (ablation) | Qwen3-30B-A3B · WikiText-2 | perplexity: 8.70 -> 59.86 | Hurt |
| prune final-layer outlier experts instead of SEs (ablation) | Qwen3-30B-A3B · WikiText-2 | perplexity: 8.70 -> 8.71 | No clear effect |
| prune final-layer outlier experts instead of SEs (ablation) | DeepSeek-V2-Lite · WikiText-2 | perplexity: 6.31 -> 6.32 | No clear effect |
| prune SEs to remove massive activations (ablation) | Qwen3-30B-A3B · attention sinks on C4 | attention sink decay rate: approximately or even exceeding 90% | Hurt |
| change SE identification threshold (P95 to P99.9, 0.07 to 0.1 of max) (ablation) | SE profiling · Qwen3-30B-A3B, DeepSeek-V2-Lite, Mixtral on C4 and WikiText-2 | number of identified SEs: 3, 2 and 1 SEs identified at all thresholds | No clear effect |
| switch dataset used for SE profiling (C4, WikiText-2, C-Eval, GSM8K, HumanEval, OoD) (ablation) | SE profiling · Qwen3-30B-A3B and others | SE distribution | No clear effect |
| compare base vs post-trained checkpoints (Base, Instruct-2507, Thinking-2507) (ablation) | Qwen3-30B-A3B · SE locations | SE set | No clear effect |
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.92)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.92)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.92)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.91)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.91)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
