AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

Ziyu Zhao, Tong Zhu, Xin Yu, Zhi Zhang, Tiantian Fan, Jinluan Yang, Kun Kuang, zhongyu wei, Fei Wu, Yu Cheng

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use multi-task Flan-v2 calibration instead of C4 (ablation)ExpertWeaver · 5-benchmark avg, Qwen2.5-7B 25% sparsityaverage accuracy: 65.8 -> 67.0Improved
increase calibration tasks from 25% to 50% of Flan-v2 (ablation)ExpertWeaver · Qwen2.5-7B 25% sparsity avgaverage accuracy: 63.9 -> 66.1Improved
increase calibration tasks from 50% to 100% of Flan-v2 (ablation)ExpertWeaver · Qwen2.5-7B 25% sparsity avgaverage accuracy: 66.1 -> 67.0Improved
increase calibration tasks from 50% to 75% (ablation)ExpertWeaver · Qwen2.5-7B 25% sparsity avgaverage accuracy: 66.1 -> 65.6Hurt
layer-aware dynamic shared-expert allocation instead of uniform ratio (ablation)ExpertWeaver · MMLU, Qwen2.5-7BMMLU accuracyImproved
use AbsTopk-GLU gating at 50% sparsity instead of WandaLLaMA3-8B · 5-benchmark avgaverage accuracy: 66.3 -> 68.2Improved
use AbsTopk-GLU instead of magnitude pruningLLaMA3-8B 50% sparsity · 5-benchmark avgaverage accuracy: 49.6 -> 68.2Improved
use ExpertWeaver instead of CMoELLaMA3-8B 25% sparsity · 5-benchmark avgaverage accuracy: 57.2 -> 60.4Improved
use ExpertWeaver instead of LLM-PrunerQwen2.5-7B 25% sparsity · 5-benchmark avgaverage accuracy: 65.0 -> 67.0Improved
use ExpertWeaver instead of LLM-Pruner on GSM8KQwen2.5-7B 25% sparsity · GSM8Kaccuracy: 2.0 -> 34.9Improved
downcycle Qwen2.5-7B with ExpertWeaver vs OLMoE-1B-7B* trained from scratchMoE 200B tokens · 8-benchmark avgaverage accuracy: 58.9 -> 63.5Improved
ExpertWeaver downcycling vs sparse upcycling and from-scratch trainingOLMo-1.3B · training loss and downstream performanceloss/accuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy