Research map / Architectures and efficiency / Mixture of experts
Exploring the Benefit of Activation Sparsity in Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Switchable Sparse-Dense Learning (SSD), which alternates dense pre-training with MoE-based sparse training (dense-to-SMoE conversion via neuron clustering) to cut pre-training FLOPs while matching dense performance and enabling 2x faster sparse inference.
Problem
Activation sparsity in pre-trained Transformers has been exploited only for post-training inference; its use during pre-training is unexplored, and sparse (MoE) pre-training usually underperforms dense training at equal parameters.
Approach
Track activation sparsity and activation-pattern similarity (via MoEfication neuron clustering and ARI) during pre-training, and switch between dense training and MoE-based sparse training, converting dense FFNs into SMoE experts with checkpoint-initialized balanced k-means and reverting to dense multiple times so the final model is dense and also usable as an SMoE for sparse inference.
Key results
- SSD matches or slightly exceeds dense training on downstream NLU (e.g., BERT avg 83.6 vs dense 83.2) with up to 1.44x FLOPs speedup in pre-training.
- BERT trained with SSD reaches dense-model performance with less than half the FFN inference time (2.3ms vs 5.1ms) using sparse inference on an RTX 3090.
- On Persimmon-8B continual pre-training, SSD reaches 32.0 C-Eval average vs 31.5 for dense training, with 1.32x FLOPs speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSD instead of dense training | BERT-base · BERT natural language understanding (Avg.) | accuracy/F1: 83.2 -> 83.6 | Helped |
| Use SSD instead of dense training | Persimmon-8B · C-Eval | accuracy: 31.5 -> 32.0 | Helped |
| Train with SSD rather than dense training | GPT-base · GPT pre-training | FLOPs: 237T -> 178T (1.44$\times$) | Helped |
| Train with SSD rather than dense training | Persimmon-8B · Persimmon-8B continual pre-training | time: 166h -> 134h (1.23$\times$) | Helped |
| Replace threshold-based transitions with random switching (ablation) | SSD on GPT-base · GPT pre-training | perplexity: 16.0 vs. 17.3; 18.6; 19.8 | Hurt |
| Randomly group neurons instead of clustering them (ablation) | SSD on GPT-base · GPT pre-training transition | perplexity: 20.4 vs. 3533.3 | Hurt |
| Remove previous-checkpoint initialization from clustering (ablation) | SSD on GPT-base · GPT pre-training transition | perplexity: 20.4 -> 20.9 | Hurt |
| Raise the conversion threshold to 0.98 (ablation) | SSD on GPT-base · GPT pre-training | perplexity | Hurt |
| Increase the sparse-mode ratio to 70% (ablation) | SSD on GPT-base · GPT pre-training | perplexity | No clear effect |
| Monitor activation patterns every 2000 steps (ablation) | SSD on GPT-base · GPT pre-training | perplexity | Hurt |
| Truncate 75% of candidate experts using importance scores instead of a fixed expert count (ablation) | SSD · Validation corpus | perplexity | Helped |
Surprising findings
- Plain SMoE pre-training consistently yields higher perplexity than dense training at equal parameters, while SSD does not.
- Dense-pretrained models perform worse under sparse inference than models pre-trained with some sparse computation, showing sparse computation during pre-training is necessary for sparse inference.
Limitations
- Activation pattern study limited to ~100M-parameter models due to compute constraints
- Actual wall-clock speedup is smaller than theoretical FLOPs speedup because expert selection adds overhead
- Pre-training uses a 27GB subset of the Pile with base-size models; GLU-based models (e.g., SwiGLU) not evaluated
Experimental setup
Datasets: The Pile SST-2 SNLI MNLI QNLI QQP SQuAD v1.1 Unnatural Instructions Super-NaturalInstructions Chinese corpus (encyclopedia, news, books, web)
Benchmarks: SST-2 SNLI MNLI QNLI QQP SQuAD v1.1 Super-NaturalInstructions C-Eval
Metrics: perplexity accuracy F1 Rouge-L FLOPs inference time
Baselines: Dense SMoE Progressive Layer Dropping MoE-Dropout MoEfication
Builds on: MoEfication MoE-Dropout Progressive Layer Dropping MegaBlocks DejaVu
Closest papers by meaning
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.88)
- Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion (NeurIPS 2024 · similarity 0.88)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.87)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.86)
- From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers (AAAI 2025 · similarity 0.86)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.