Research map / Time series / Long-term forecasting
Scaling transformer neural networks for skillful and reliable medium-range weather forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stormer, a standard ViT-based weather forecaster trained with a randomized dynamics forecasting objective, pressure-weighted loss, and weather-specific embedding, which beats GraphCast and Pangu-Weather beyond 7 days on WeatherBench 2 with far less data and compute.
Problem
Existing data-driven weather forecasting models use complex, customized architectures with little ablation, so it is unclear which components drive their skill; they also require heavy training compute and high-resolution data.
Approach
A simple ViT with weather-specific embedding (variable tokenization plus cross-attention variable aggregation), adaLN conditioning on the time interval, and a randomized-interval dynamics forecasting objective; a single model yields multiple forecasts for a lead time by combining different interval roll-outs and averaging them.
Key results
- At 14-day lead time Stormer beats GraphCast by 10%-20% RMSE across nine key variables, and it is the only model that beats climatology at long lead times.
- Stormer trained on 1.40625-degree data for under 24 hours on 128 A100 GPUs, roughly 190x less data than Pangu-Weather and 90x less than GraphCast.
- Randomized-interval training with best m-in-n inference consistently outperforms single-interval models at all lead times, with the gap widening over time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use randomized interval training and combine forecasts from multiple interval sequences instead of a single fixed interval (ablation) | Stormer · WeatherBench 2 surface temperature, 1–14-day forecasts | RMSE | Helped |
| Use pressure-weighted loss instead of unweighted loss (ablation) | Stormer · WeatherBench 2 | RMSE | Helped |
| Predict weather dynamics rather than complete future weather states (ablation) | Stormer · WeatherBench 2 | RMSE | Helped |
| Use multi-step fine-tuning with K=8 rather than the first phase with K=1 (ablation) | Stormer · WeatherBench 2 long-lead forecasts | RMSE | Helped |
| Increase model size from Stormer-S to Stormer-B to Stormer-L (ablation) | Stormer · WeatherBench 2 | RMSE | Helped |
| Reduce patch size, increasing the number of training tokens (ablation) | Stormer · WeatherBench 2 | RMSE | Helped |
| Use weather-specific embedding instead of standard patch embedding (ablation) | Stormer · WeatherBench 2, 1–10-day forecasts | RMSE | Helped |
| Condition transformer blocks with adaptive layer normalization instead of additive lead-time embedding (ablation) | Stormer · WeatherBench 2 | RMSE | Helped |
| Randomize the time interval separately at each rollout step during multi-step fine-tuning (ablation) | Stormer · WeatherBench 2 | forecast performance | Hurt |
| Use best m in n inference rather than homogeneous inference (ablation) | Stormer · WeatherBench 2 | RMSE | No clear effect |
| Add initial-condition perturbations to ensemble members (ablation) | Stormer · WeatherBench 2 probabilistic forecasts | probabilistic metric | Helped |
| Add initial-condition perturbations to ensemble members (ablation) | Stormer · WeatherBench 2 short-lead forecasts | deterministic performance | Hurt |
Surprising findings
- A plain ViT with proper training recipe matches or beats specialized architectures like GraphCast and Pangu-Weather at long lead times despite lower resolution data.
- Forecasting dynamics (differences) is consistently more accurate than forecasting absolute weather states.
Limitations
- Trained only on 1.40625-degree data; higher resolution and larger models left to future work.
- Ensemble-style averaging of multiple roll-outs is used; non-ensemble version lags GraphCast at short range, and IC-perturbed forecasts are underdispersive.
- Lags GraphCast in short-range (1-5 day) forecast accuracy and lags IFS HRES for T2m at initial lead times.
Experimental setup
Datasets: ERA5 (WeatherBench 2)
Benchmarks: WeatherBench 2
Metrics: latitude-weighted RMSE ACC
Baselines: Pangu-Weather GraphCast Climatology IFS HRES IFS ENS
Builds on: Vision Transformer GraphCast Pangu-Weather ClimaX FourCastNet
Closest papers by meaning
- OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scales (NeurIPS 2025 · similarity 0.87)
- SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.87)
- Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling (NeurIPS 2024 · similarity 0.85)
- TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting (NeurIPS 2025 · similarity 0.85)
- Loss Shaping Constraints for Long-Term Time Series Forecasting (ICML 2024 · similarity 0.85)
- Unified Training of Universal Time Series Forecasting Transformers (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.