Research map / Probabilistic and scientific modeling / Molecules and proteins
ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChaosBench, a multi-source (ERA5, ORAS5, LRA5) benchmark with physics-based and probabilistic metrics for evaluating data-driven and physics-based models on subseasonal-to-seasonal (S2S) climate prediction up to 44 days.
Problem
Data-driven weather emulators are evaluated mainly at short/medium range (up to 15 days) or climate scales, and S2S forecasting (sensitive to both initial and boundary conditions) lacks a benchmark with diverse Earth-system data, operational baselines, and physics-based metrics.
Approach
Combine multi-source Earth-system reanalysis (ERA5, ORAS5, LRA5) spanning 45+ years with 44-day forecasts from four national weather agencies as baselines, plus spectral-divergence and spectral-residual physics metrics and probabilistic ensemble metrics, to test whether data-driven emulators extend predictability to S2S timescales.
Key results
- At 44-day lead, data-driven autoregressive models PanguWeather, GraphCast and FourCastNetV2 perform worse than climatology on t-850 (RMSE 5.85, 5.87, 5.11 vs 3.39 K), while direct ViT/ClimaX achieves 3.56 K.
- Physics-based ECMWF high-resolution ensemble has the best CRPSS, with predictability around 15-20 days before skill collapses to climatology.
- Ensemble forecasts improve over deterministic ones at longer lead times, with RMSE ratio decreasing with lead time and benefiting from more ensemble members.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use direct rather than autoregressive forecasting | data-driven weather emulators · S2S forecasts, Δt=44 | RMSE: 5.85 -> 3.56; 5.87 -> 3.56; 5.11 -> 3.56 | Helped |
| increase ensemble size | physics-based forecast models · S2S forecasts | forecast skill | Helped |
| use ensemble forecasts instead of deterministic forecasts | physics-based forecast models · S2S forecasts | RMSE | Helped |
| incorporate physical knowledge, such as learning spectral signals | data-driven weather emulators · S2S forecasts | performance across metrics | Helped |
| use autoregressive rather than direct forecasting | data-driven weather emulators · S2S forecasts, Δt=44 | RMSE: 3.56 -> 5.85; 3.56 -> 5.87; 3.56 -> 5.11 | Hurt |
Surprising findings
- Most physics-based control forecasts and deterministic data-driven models perform worse than climatology beyond about 15 days on S2S timescales.
- Direct forecasting (ViT/ClimaX) outperforms autoregressive models at 44 days, indicating error propagation is a major source of error.
Limitations
- Input datasets are at coarse ~1.5 degree resolution to match physics-based forecasts; higher resolution requires user-side processing.
- Physics-based forecast archive covers only 2016 onward, limiting the evaluation period.
- Plans to add multi-source reanalysis such as MERRA-2 are future work; current benchmark uses a limited set of reanalyses.
Experimental setup
Datasets: ERA5 ORAS5 LRA5 UKMO GloSea6 forecasts NCEP CFSv2 forecasts CMA BCC-CSM2-HR forecasts ECMWF IFS forecasts
Benchmarks: ChaosBench
Metrics: RMSE Bias ACC MS-SSIM SpecDiv SpecRes CRPS CRPSS Spread Spread/Skill ratio
Baselines: Climatology Persistence PanguWeather GraphCast FourCastNetV2 ViT/ClimaX UKMO GloSea6 NCEP CFSv2 CMA BCC-CSM2-HR ECMWF IFS
Builds on: WeatherBench ClimaX GraphCast FourCastNetV2 PanguWeather
Closest papers by meaning
- SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarking (NeurIPS 2023 · similarity 0.89)
- OceanBench: A Benchmark for Data-Driven Global Ocean Forecasting systems (NeurIPS 2025 · similarity 0.89)
- ClimSim: A large multi-scale dataset for hybrid physics-ML climate emulation (NeurIPS 2023 · similarity 0.88)
- ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs (ICLR 2024 · similarity 0.86)
- OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scales (NeurIPS 2025 · similarity 0.85)
- Scaling Storm-Resolving Atmospheric AI Simulation to the Entire Planet (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.