AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

SD-MoE: Spectral Decomposition for Effective Expert Specialization

Ruijun Huang, Fang Dong, Xin Zhang, Anrui Chen, Hengjie Cao, Zhendong Huang, Jixian Zhou, Mengyi Chen, Yifeng Yang, Mingzhi Dong, Yujiang Wang, Jinlong Hou and 7 more

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
apply SD-MoE spectral decomposition to Qwen MoEQwen-2B-A0.8B · 8 downstream tasks avgaverage accuracy (%): 50.45 -> 53.97Improved
apply SD-MoE to DeepSeek MoEDeepSeek-2B-A0.8B · 8 downstream tasks avgaverage accuracy (%): 50.51 -> 51.49Improved
apply SD-MoE to larger Qwen MoEQwen-7B-A1.5B · 8 downstream tasks avgaverage accuracy (%): 55.34 -> 57.23Improved
apply SD-MoE (LAMBADA exception at large scale)Qwen-7B-A1.5B · LAMBADA_OPENAIaccuracy (%): 49.31 -> 47.29Hurt
add gradient projection to parameter decomposition (ablation)SD-MoE · 8 downstream tasks avgaverage accuracy (%): 48.54 -> 48.86 (baseline); full method needed for gainImproved
vary common subspace rank k in {2,4,8,16,32} (ablation)SD-MoE · 8 downstream tasks avgaverage accuracy (%): 53.64, 53.39, 53.97, 53.57, 53.68No clear effect
use SD-MoE instead of AES diversity regularizationQwen-2B-A0.8B · 8 downstream tasks avgaverage accuracy (%): 51.58 -> 53.97Improved
apply SD-MoE for summarizationQwen MoE · XSUMROUGE-1: 33.10 -> 34.27Improved
apply SD-MoE to allow larger learning rateQwen MoE · learning rate stabilitymax stable learning rate: 1e-4 -> 4e-4Improved
apply SD-MoE (training cost)Qwen-2B-A0.8B · training throughputtokens/sec: 229K -> 218KHurt
apply SD-MoE to reduce inter-expert subspace similarityQwen MoE · expert principal subspace similaritysimilarity: ~0.7 avg -> ~0.1Improved
periodic SVD refresh interval 8 vs 16 vs 32 steps (ablation)SD-MoE · training overheadSVD refresh overhead: 0.52% / 0.46% / 0.35%No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy