Research map / Architectures and efficiency / Mixture of experts
EWMoE: An Effective Model for Global Weather Forecasting with Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EWMoE, a ViT-based global weather forecasting model with 3D absolute position embedding, a 20-expert top-2 MoE layer, and auxiliary plus position-weighted losses, trained on only two years of ERA5 data.
Problem
Data-driven weather forecasting models such as FourCastNet, ClimaX, Pangu-Weather and GraphCast need massive training data and compute; the goal is accurate global forecasting with far less data and compute.
Approach
Combines a ViT backbone with a 3D absolute (longitude, latitude, altitude) position embedding and a sparsely activated Mixture-of-Experts layer that adds capacity without extra per-token compute, plus a load-balancing auxiliary loss and a latitude/position-weighted MSE loss.
Key results
- EWMoE outperforms FourCastNet and ClimaX at all lead times on ACC and RMSE for the analyzed variables (Z500, T2m, T850, U10).
- EWMoE is comparable to Pangu-Weather for 1-3 day forecasts and outperforms it from day 3; versus GraphCast it has higher ACC but slightly worse RMSE.
- Training uses about 2 years of ERA5 data (~18x less than FourCastNet/ClimaX) and completed in 9 days on 2 Nvidia 3090 GPUs; the MoE layer raises parameters from 43M to 580M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EWMoE instead of FourCastNet | Global weather forecasting on ERA5 · Z500, T2m, T850, U10; forecasts from 1 to 8 days | ACC | Helped |
| Use EWMoE instead of ClimaX | Global weather forecasting on ERA5 · Z500, T2m, T850, U10; forecasts from 1 to 8 days | ACC | Helped |
| Use EWMoE instead of Pangu-Weather | Global weather forecasting on ERA5 · Z500, T2m, T850, U10 | ACC | Mixed |
| Use EWMoE instead of GraphCast | Global weather forecasting on ERA5 · Z500, T2m, T850, U10 | ACC | Helped |
| Use EWMoE instead of GraphCast | Global weather forecasting on ERA5 · Z500, T2m, T850, U10 | RMSE | Hurt |
| Add 3D absolute position embedding instead of standard ViT position embedding (ablation) | EWMoE · ERA5 weather forecasting, all forecast times | forecast performance | Helped |
| Add an MoE layer instead of a standard feed-forward network (ablation) | EWMoE · ERA5 weather forecasting | forecast performance | Helped |
| Add an MoE layer instead of a standard feed-forward network (ablation) | EWMoE · ERA5 weather forecasting at longer forecast times | forecast performance | Helped |
| Add auxiliary loss for MoE routing (ablation) | EWMoE · ERA5 weather forecasting | forecast performance | Helped |
| Use smaller top-k for expert selection (ablation) | EWMoE · ERA5 weather forecasting | forecast performance | Helped |
Surprising findings
- EWMoE has higher ACC than GraphCast but slightly worse RMSE, indicating mixed results against the strongest baseline.
Limitations
- Trails GraphCast on RMSE and does not use GraphCast's autoregressive finetuning.
- Evaluated on a single test year (2018) and four key variables only.
- Uses only 20 variables across five vertical levels, a reduced input set relative to the baselines.
Experimental setup
Datasets: ERA5
Benchmarks: ERA5 2018 test year
Metrics: latitude-weighted ACC latitude-weighted RMSE
Baselines: FourCastNet ClimaX Pangu-Weather GraphCast
Builds on: Vision Transformer ClimaX Mixture-of-Experts FourCastNet Stormer
Closest papers by meaning
- Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models (NeurIPS 2024 · similarity 0.85)
- Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (ICLR 2025 · similarity 0.84)
- Scaling transformer neural networks for skillful and reliable medium-range weather forecasting (NeurIPS 2024 · similarity 0.84)
- OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scales (NeurIPS 2025 · similarity 0.84)
- Probabilistic Weather Forecasting with Hierarchical Graph Neural Networks (NeurIPS 2024 · similarity 0.84)
- WeatherGFM: Learning a Weather Generalist Foundation Model via In-context Learning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.