Research map / Architectures and efficiency / Mixture of experts
Eigenvectors of Experts are Training-free Non-collapsing Routers
Giang Do, Hung Le, Truyen Tran
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace learned router with SSMoE eigenvector routing, with 25% experts dropped | GPT-OSS-20B · 10-shot 8-benchmark average | average accuracy (%): 48.8 -> 54.1 vs original | Improved |
| replace learned router with SSMoE eigenvector routing, with 25% experts dropped | GPT-OSS-120B · 10-shot 8-benchmark average | average accuracy (%): 52.6 -> 54.9 vs original | Improved |
| use SSMoE to choose which experts to drop instead of random drop (ablation) | GPT-OSS-20B · 10-shot 8-benchmark average | average accuracy (%): 44.9 -> 54.1 | Improved |
| keep all experts with SSMoE routing (SSMoE-FULL) (ablation) | GPT-OSS-20B · 10-shot 8-benchmark average | average accuracy (%): 54.1 -> 55.2 | Improved |
| SSMoE routing on GSM8K with 25% experts dropped | GPT-OSS-20B · 10-shot GSM8K | accuracy (%): 36.8 -> 30.8 | Hurt |
| use eigenvector-only (EV) representation in place of learned router embeddings (ablation) | OLMoE-7B · MTEB clean average | average score: 29.9 -> 40.9 | Improved |
| add router mixing to the EV representation (SSMoE vs EV) (ablation) | OLMoE-7B · MTEB clean average | average score: 40.9 -> 39.7 | No clear effect |
| use SSMoE instead of MoEE embeddings on Qwen-MoE-7B | MoEE · MTEB clean average | average score: 42.6 -> 45.8 | Improved |
| use SSMoE under corrupted inputs | SMoE / MoEE · MTEB corrupt average | average score: 21.2 -> 24.8 (OLMoE vs MoEE) | Improved |
| apply SSMoE routing to CLIP-MoE for zero-shot retrieval | CLIP-MoE · COCO I2T Recall@1 clean | Recall@1: 65.0 -> 65.7 | Improved |
| apply SSMoE routing to CLIP-MoE with noisy images | CLIP-MoE · COCO I2T Recall@1 corrupt | Recall@1: 36.2 -> 37.2 | Improved |
| use averaged eigenvector descriptors instead of router-guided selection (ablation) | SSMoE · GPT-OSS-20B 10-shot average | average accuracy (%): 54.1 -> 52.7 | Hurt |
Closest papers by meaning
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.90)
- Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts (NeurIPS 2026 · similarity 0.89)
- Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · similarity 0.89)
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024 · similarity 0.89)
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.88)
- Tight Clusters Make Specialized Experts (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
