AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Less Token, More Signal: MoE Expert Pruning via Critical Token Selection

Zeliang Zong, Kai Zhang, Yarong Wang, Wenming Tan, Ye Ren, Jilin Hu

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add attention-guided token selection to frequency-based pruning (ablation)frequency-based expert pruning on OLMoE-7A1B · 8 zero-shot tasks avg (25% pruned)accuracy: 53.48 -> 55.12Improved
add dual-factor scoring (ablation)OLMoE-7A1B pruning · 8 zero-shot tasks avg (25% pruned)accuracy: 53.48 -> 55.43Improved
add experts-to-bias conversion on top of dual-factor scoring (ablation)OLMoE-7A1B pruning · 8 zero-shot tasks avg (25% pruned)accuracy: 55.43 -> 56.16Improved
add token selection on top of dual-factor scoring and experts-to-bias (ablation)OLMoE-7A1B pruning · 8 zero-shot tasks avg (25% pruned)accuracy: 56.16 -> 58.42 (+2.26)Improved
add experts-to-bias stage (Stage 3) to STEP (ablation)STEP on OLMoE-7A1B · 8 zero-shot tasks avg (25% pruned)accuracy: 57.51 -> 58.42Improved
add experts-to-bias stage (Stage 3) to STEP (ablation)STEP on OLMoE-7A1B · 8 zero-shot tasks avg (50% pruned)accuracy: 48.31 -> 50.05Improved
add experts-to-bias stage (Stage 3) to STEP (ablation)STEP on Qwen3-30A3B · 8 zero-shot tasks avg (25% pruned)accuracy: 68.84 -> 69.06No clear effect
reduce token retention from 100% to 50% (ablation)STEP on OLMoE-7A1B · WikiText perplexity (25% pruned)perplexity: 11.65 -> 10.40Improved
reduce token retention from 50% to 20% (ablation)STEP on OLMoE-7A1B · 8 zero-shot tasks avg (25% pruned)accuracy: 58.42 -> 58.81No clear effect
use gradient-based token importance instead of attention (ablation)STEP token selection · 8 zero-shot tasks avg (OLMoE-7A1B)accuracy gain: +0.42 and +0.24 vs +2.26 and +2.65No clear effect
use L2 norm or update magnitude token scoring instead of attention (ablation)STEP token selection · 8 zero-shot tasks avg (OLMoE-7A1B)accuracy gain: -3.06, -4.54, -2.72, -3.09Hurt
use LLMLingua token selection instead of attention (ablation)STEP token selection · 8 zero-shot tasks avg (OLMoE-7A1B)accuracy: 58.81 -> 57.22Hurt

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy