Research map / Vision-language and multimodal / Multimodal LLMs

WeatherSyn: An Instruction Tuning MLLM For Weather Forecasting Report Generation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Weather Forecasting Report (WFR) task and the WSInstruct dataset of ERA5 heatmaps paired with expert forecast reports for 31 US cities, then fine-tunes Qwen3-VL-8B (SFT, rejection-sampling FT, DPO) into WeatherSyn, which outperforms closed-source MLLMs on claim-based F1.

Problem

Generating human-readable, factually accurate multi-day weather forecast reports directly from multi-variable meteorological visual inputs; existing weather MLLMs focus on classification, QA or severe-weather alerts rather than general forecast reports.

Approach

Builds an instruction-tuning dataset pairing regional ERA5 variable heatmaps with expert forecast discussions split into daily reports, uses aspect-controlled prompting and claim-based evaluation, and improves lexical diversity via step-level rejection sampling followed by DPO on F1-ranked preference pairs.

Key results

• WeatherSyn-DPO reaches weighted F1 0.59 average across 8 aspects versus 0.60 for Gemini-3 Pro Preview (2-shot) and 0.49 for GPT-5.2 Thinking; WeatherSyn (SFT) reaches 0.55.
• On the reference-based metrics, WeatherSyn-DPO gets BLEU-1 0.44 and ROUGE-L 0.32, above all closed-source models.
• LLM-judge factual consistency Top-1 rate rises from 0.03-0.06 for closed-source GPT models to 0.33 for WeatherSyn-DPO (ranking-based protocol).

Surprising findings

• Reference-based metrics (BLEU, ROUGE, METEOR) favor open-source baselines such as WeatherQA, contradicting the claim-based and LLM/expert rankings, indicating surface metrics are inadequate for this task.
• DPO gains are limited and aspect-dependent, with performance degradation on some aspects.

Limitations

• Data restricted to 31 US cities and reports from a single source (AFD products), limiting geographic generalization.
• Uses only the initial atmospheric state as input; no NWP forecast fields in the main setting.
• Evaluation relies on LLM-based claim extraction (Qwen-2.5-72B) and small LLM/expert judge sets (144 reports).

Experimental setup

Datasets: WSInstruct WSInstruct-RFT WSInstruct-DPO ERA5 Iowa Environmental Mesonet AFD reports WeatherQA public data OmniEarth public data

Benchmarks: WSInstruct test set

Metrics: BLEU-1 ROUGE-L METEOR weighted precision weighted F1 factual consistency Top-1 rate summarization quality Top-1 rate hit rate

Baselines: GPT-4.1-Mini GPT-5-Nano GPT-5.2 Thinking Gemini-2.5-Flash Gemini-3 Pro Preview Claude-3.7-Sonnet WeatherQA OmniEarth

Builds on: Qwen3-VL direct preference optimization WeatherQA OmniEarth-Bench

Closest papers by meaning

• RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts (NeurIPS 2025 · similarity 0.87)
• Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models (NeurIPS 2024 · similarity 0.87)
• WeatherGFM: Learning a Weather Generalist Foundation Model via In-context Learning (ICLR 2025 · similarity 0.86)
• Scaling transformer neural networks for skillful and reliable medium-range weather forecasting (NeurIPS 2024 · similarity 0.85)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information (ICML 2025 · similarity 0.83)
• ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities (ICLR 2025 · similarity 0.83)