Research map / Architectures and efficiency / Mixture of experts
A Theoretical View on Sparsely Activated Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace dense layer with DSM 25% sparsity at 73728 FLOPs | 2-layer network · random polynomial | eval MSE: 0.01046 -> 0.001799 | Helped |
| replace dense layer with DSM 50% sparsity at 73728 FLOPs | 2-layer network · random polynomial | eval MSE: 0.01046 -> 0.006115 | Helped |
| use DSM 50% sparsity at lowest FLOPs | 2-layer network · random polynomial | eval MSE: 0.01015 -> 0.01014 | No clear effect |
| use DSM 25% sparsity at 18432 FLOPs | 2-layer network · random polynomial | eval MSE: 0.01015 -> 0.009655 | Helped |
| use data-dependent sparsity (DSM/LSH) instead of dense with same activated units | DSM, LSH · random polynomial and hypercube | MSE | Helped |
| use random hash sparsity instead of data-dependent sparsity | random hash sparse model · random polynomial | MSE | Hurt |
| use LSH with fewer than 2048 buckets (ablation) | LSH · random polynomial | MSE | Hurt |
| replace penultimate dense layer with Top-K DSM layer | CNN + dense · CIFAR-10 | accuracy | Helped |
Closest papers by meaning
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.89)
- Exploring the Benefit of Activation Sparsity in Pre-training (ICML 2024 · similarity 0.88)
- The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis (NeurIPS 2025 · similarity 0.87)
- No Free Prune: Information-Theoretic Barriers to Pruning at Initialization (ICML 2024 · similarity 0.87)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.86)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.