Research map / Architectures and efficiency / Mixture of experts
Spatial Mixture-of-Experts
Nikoli Dryden, Torsten Hoefler
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace all 3×3 convolutions with SMoE layers and add a land-sea gate prior | ResNet on WeatherBench · WeatherBench | latitude-weighted RMSE: 316 -> 270; 563 -> 525; 1.80 -> 1.66; 2.84 -> 2.60 | Improved |
| Replace U-Net 3×3 convolutions with SMoE layers | U-Net · ENS-10 | CRPS: 76.54 -> 68.94; 0.685 -> 0.612; 0.657 -> 0.601 | Improved |
| Replace all 3×3 convolutions with SMoE layers | ResNet-50 · ImageNet-1k val | Top-1 accuracy: 80.83 -> 81.33 | Improved |
| Add routing classification loss (ablation) | SMoE · Heat diffusion | % within 1%: 91.5 -> 96.7 | Improved |
| Add expert error damping to SMoE with routing classification loss (ablation) | SMoE · Heat diffusion | % within 1%: 96.7 -> 100.0 | Improved |
| Add expert error damping without routing classification loss (ablation) | SMoE · Heat diffusion | % within 1%: 91.5 -> 91.6 | No clear effect |
| Use tensor routing instead of a fully-connected, convolutional, CoordConv, or LCN gate (ablation) | SMoE · Heat diffusion | % within 1%: 85.2 -> 100.0; 91.3 -> 100.0; 91.5 -> 100.0; 96.3 -> 100.0 | Improved |
| Add a land-sea prior to the learned gate (ablation) | SMoE · WeatherBench | latitude-weighted RMSE: 278 -> 270; 530 -> 525; 1.69 -> 1.66; 2.65 -> 2.60 | Improved |
| Fix a randomly initialized gate instead of learning the routing (ablation) | SMoE · WeatherBench | latitude-weighted RMSE: 278 -> 328; 530 -> 572; 1.69 -> 1.89; 2.65 -> 2.96 | Hurt |
| Add routing noise (ablation) | SMoE with RC loss and damping · Heat diffusion | % within 1%: 100.0 -> 95.7 | Hurt |
| Replace convolution experts with CoordConv experts (ablation) | SMoE · Heat diffusion | % within 1%: 100.0 -> 96.5 | Hurt |
| Remove gate-based expert-output weighting (ablation) | SMoE · Heat diffusion | % within 1%: 100.0 -> 100.0 | No clear effect |
Closest papers by meaning
- On the Integration of Spatial-Temporal Knowledge: A Lightweight Approach to Atmospheric Time Series Forecasting (NeurIPS 2025 · similarity 0.85)
- ST-FiT: Inductive Spatial-Temporal Forecasting with Limited Training Data (AAAI 2025 · similarity 0.83)
- Explanation of Dynamic Physical Field Predictions using WassersteinGrad: Application to Autoregressive Weather Forecasting (NeurIPS 2026 · similarity 0.83)
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024 · similarity 0.83)
- MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts (NeurIPS 2024 · similarity 0.83)
- ST-ReP: Learning Predictive Representations Efficiently for Spatial-Temporal Forecasting (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
