Research map / Architectures and efficiency / Mixture of experts
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Top-k gate with DSelect-k gate (per-example) | multi-gate MoE · MovieLens alpha=0.9 | test loss: 3741 -> 3679 | Helped |
| replace Top-k gate with DSelect-k gate (static) | multi-gate MoE · MovieLens alpha=0.1 | test loss: 4012 -> 4015 | No clear effect |
| replace Top-k with DSelect-k (static) | multi-gate MoE · Multi-MNIST Task 2 | accuracy: 90.03 -> 90.98 | Helped |
| replace Top-k with DSelect-k (static) | multi-gate MoE · Multi-Fashion Task 2 | accuracy: 82.66 -> 83.34 | Helped |
| use sparse gate (Top-k or DSelect-k) instead of dense softmax over all experts | multi-gate MoE · MovieLens alpha=0.5 | test loss: 3960 -> 3804 | Helped |
| switch DSelect-k from static to per-example gating | DSelect-k · Multi-MNIST | accuracy: 90.98 -> 90.7 | No clear effect |
| replace binary-encoding selector with softmax selector plus temperature annealing (ablation) | DSelect-k · Synthetic MoE data | MSE and Jaccard index | Hurt |
| replace binary-encoding selector with softmax selector plus entropy regularization (ablation) | DSelect-k · Synthetic MoE data | experts selected (k=4): 4 -> ~2 | Hurt |
| replace Top-k with DSelect-k at 128 tasks | multi-gate MoE · Synthetic 128 tasks | MSE: over 40% improvement | Helped |
| replace Top-k with DSelect-k on large-scale recommender | multi-task ranking MoE · Recommender system 8 tasks | AUC/RMSE: over 22% improvement | Helped |
| replace Top-k with DSelect-k for expert recovery with frozen experts | gate only · Synthetic 4-of-16 expert recovery | experts recovered: 1 -> 4 | Helped |
| DSelect-k versus Top-k training compute | MoE training · MovieLens alpha=0.1 | FLOPS ratio DSelect-k/Top-k: 1.5 vs 0.6 at alpha=0.9 | Mixed |
Closest papers by meaning
- Multi-Task Dense Prediction via Mixture of Low-Rank Experts (CVPR 2024 · similarity 0.87)
- Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning (ICML 2025 · similarity 0.87)
- Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.87)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.87)
- Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion (NeurIPS 2024 · similarity 0.87)
- Learning Mixtures of Experts with EM: A Mirror Descent Perspective (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.