Research map / Vision-language and multimodal / Multimodal LLMs
WeatherGFM: Learning a Weather Generalist Foundation Model via In-context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WeatherGFM, a ViT-based weather generalist foundation model trained with masked visual-prompt question-answering to handle ten weather understanding tasks and generalize in-context to unseen tasks.
Problem
Existing data-driven weather models address single weather understanding tasks with limited data and cannot handle diverse tasks and data modalities within one unified model.
Approach
Unifies weather tasks as visual prompt question-answer pairs with weather-specific prompt formats (single, cross-modal, time-series) and trains a ViT with mixed-modal masked image modeling so that a single model performs many tasks via in-context learning, including unseen ones.
Key results
- WeatherGFM (ten tasks, single model) matches or beats single-task ViT on most reported tasks, e.g., Radar Temporal SR RMSE 0.327 vs 0.333 for ViT and GOES2Radar RMSE 0.436 vs 0.445 for ViT-ST (in the ablation table).
- On ERA5 T2m at 168-hour lead time, WeatherGFM achieves RMSE 1.76 versus ClimaX 2.17 and IFS 2.23.
- Prompt choice matters for some tasks: CSI standard deviation across 20 prompts reaches about 0.02 for radar extrapolation and GOES2Radar, while super-resolution is nearly insensitive.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WeatherGFM instead of single-task ViT | Radar extrapolation · Radar extrapolation | RMSE: 0.490 -> 0.467 | Helped |
| Use WeatherGFM instead of single-task ViT | Radar extrapolation · Radar extrapolation | CSI/219: 0.007 -> 0.021 | Helped |
| Use WeatherGFM instead of single-task ViT | Radar spatial SR · Radar spatial SR | RMSE: 0.120 -> 0.121 | Hurt |
| Use WeatherGFM instead of single-task ViT | GOES2Radar · GOES2Radar | RMSE: 0.445 -> 0.436 | Helped |
| Use WeatherGFM instead of single-task ViT | GOES2Radar · GOES2Radar | CSI/219: 0.042 -> 0.053 | Helped |
| Use a single WeatherGFM for multiple lead times rather than separate ClimaX models | ERA5 T2m forecasting at 120 hr · ERA5 T2m, 120 hr | RMSE: 1.83 -> 1.68 | Helped |
| Use a single WeatherGFM for multiple lead times rather than separate ClimaX models | ERA5 T2m forecasting at 168 hr · ERA5 T2m, 168 hr | RMSE: 2.17 -> 1.76 | Helped |
| Train on four tasks rather than single-task ViT (ablation) | WeatherGFM-4Tasks · GOES2Radar | CSI/74: 0.424 -> 0.443 | Helped |
| Train on four tasks rather than single-task ViT (ablation) | WeatherGFM-4Tasks · Radar temporal SR | RMSE: 0.333 -> 0.353 | Hurt |
| Train on ten tasks rather than four tasks (ablation) | WeatherGFM · Radar temporal SR | RMSE: 0.353 -> 0.327 | Helped |
| Train on ten tasks rather than four tasks (ablation) | WeatherGFM · GOES2GOES | RMSE: 0.317 -> 0.310 | Helped |
| Use high-quality prompts rather than random prompts (ablation) | WeatherGFM · GOES2Radar | CSI/74: 0.389 -> 0.399 | Helped |
Surprising findings
- Performance varies noticeably with the choice of visual prompt for forecasting and translation tasks (CSI std about 0.02).
- The 4-task model underperforms single-task ViT on the satellite GOES-IR2GOES-IR task, suggesting multi-task gains depend on task similarity.
Limitations
- Performance on the hardest OOD task (multi-modal satellite spatial SR) fails to give meaningful outputs.
- Authors state the goal is not SOTA on every task; several tasks are only marginally better than or below single-task baselines.
- Evaluation is on SEVIR and a regional NO2 dataset, with limited scale and no release statement.
Experimental setup
Datasets: SEVIR POMINO-TROPOMI GEOS-CF ERA5
Benchmarks: SEVIR POMINO-TROPOMI
Metrics: RMSE CSI
Baselines: UNet ViT ClimaX IFS
Builds on: Painter MAE-VQGAN ClimaX Earthformer
Closest papers by meaning
- Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models (NeurIPS 2024 · similarity 0.87)
- WeatherSyn: An Instruction Tuning MLLM For Weather Forecasting Report Generation (ICML 2026 · similarity 0.86)
- Language-driven All-in-one Adverse Weather Removal (CVPR 2024 · similarity 0.86)
- AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science (NeurIPS 2025 · similarity 0.85)
- Semantic Segmentation in Multiple Adverse Weather Conditions with Domain Knowledge Retention (AAAI 2024 · similarity 0.85)
- OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scales (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.