Research map / Architectures and efficiency / Mixture of experts
Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts dense Transformers into MoE models by regularizing activation sparsity, training regression-based routers that predict expert output norms, and using per-token dynamic-k expert selection, extended to MHA projections and gated MLPs.
Problem
Converting pre-trained dense Transformers into efficient MoE models (MoEfication) yields a suboptimal accuracy-compute trade-off, since the base model's activation sparsity is not exploited, routers are poorly trained, and static top-k ignores large per-token variance in activated neurons.
Approach
Enforce higher activation sparsity via a square Hoyer regularizer before conversion, train routers as regressors of each expert's output L2 norm, select experts dynamically with a threshold relative to the max predicted norm, and replace standalone dense layers (including MHA projections) with small distilled MLPs.
Key results
- D2DMoE reduces inference cost by up to 60% without significantly impacting performance across NLP and vision tasks.
- On BoolQ with Gemma-2B, D2DMoE retains 98.69% relative accuracy at 70% compute budget, while MoEfication drops to 92.15%.
- A single D2DMoE layer on A100 can be almost three times as fast as the standard MLP while preserving 99% of original accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D2DMoE instead of MoEfication | Gemma-2B · BoolQ | relative accuracy: 92.24% -> 99.68%; 92.19% -> 99.37%; 92.15% -> 98.69%; 88.79% -> 97.60%; 75.40% -> 94.34%; 86.70% -> 92.75%; 77.53% -> 90 | Helped |
| Add activation sparsity enforcement before conversion (ablation) | D2DMoE · BERT-base on CARER | accuracy versus FLOPs trade-off | Helped |
| Replace the MoEfication router objective with regression-based expert-contribution prediction (ablation) | D2DMoE · BERT-base on CARER | accuracy versus FLOPs trade-off | Helped |
| Use dynamic-k expert assignment instead of fixed top-k (ablation) | D2DMoE · BERT-base on CARER | accuracy versus FLOPs trade-off | Helped |
| Convert MHA projection layers in addition to FFN layers (ablation) | D2DMoE · BERT-base on CARER | FLOPs at retained accuracy | Helped |
| Increase activation sparsity in the dense base model (ablation) | D2DMoE · GPT-2-base on OpenWebText | cost-accuracy trade-off | Helped |
| Use regression routing instead of vanilla MoEfication routing (ablation) | MoEfication-converted Gemma-2B · C4 language modeling | loss versus compute | Helped |
| Use a sparsified GELU-based model rather than a non-sparsified GELU-based model (ablation) | D2DMoE · GPT-2-base on OpenWebText | language-model performance | Helped |
| Use smaller experts (ablation) | D2DMoE · GPT-2-base on OpenWebText | cost-accuracy trade-off | Helped |
| Set expert size to 1 (ablation) | D2DMoE · GPT-2-base on OpenWebText | performance | Hurt |
| Use a 2-layer MLP router instead of a linear router (ablation) | D2DMoE · BERT-base and GPT-2-base | performance | Helped |
| Replace FFN layers only rather than also replacing MHA projections (ablation) | D2DMoE · BERT-base on CARER | FLOPs at retained accuracy | Hurt |
Surprising findings
- MoEfication's original router-training code differs from the objective described in its paper (binary cross-entropy on batch-normalized activation sums).
- MoEfication collapses on Gemma-2B, attributed to massive activations corrupting its normalized router labels; replacing only the router with regression routing substantially improves it even without sparsification.
Limitations
- Sparsity enforcement and router training add extra training time overhead.
- Experiments limited by compute; not tested on extremely large models.
- Language modeling results reported without MHA projection replacement because it is sensitive to attention changes.
Experimental setup
Datasets: ImageNet-1k CARER OpenWebText C4 BoolQ QNLI Wikipedia BooksCorpus
Benchmarks: ImageNet-1k CARER OpenWebText C4 BoolQ QNLI
Metrics: accuracy test loss FLOPs wall-clock time relative accuracy
Baselines: MoEfication Zero-time Waste (ZTW) A-ViT CoFi relufication
Builds on: MoEfication relufication Hoyer sparsity regularization Zero-time Waste A-ViT
Closest papers by meaning
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.88)
- From Sparse to Soft Mixtures of Experts (ICLR 2024 · similarity 0.88)
- Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks (CVPR 2025 · similarity 0.88)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.88)
- Exploring the Benefit of Activation Sparsity in Pre-training (ICML 2024 · similarity 0.88)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.