Research map / Vision-language and multimodal / Multimodal LLMs
RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RadarQA, an MLLM (Qwen2.5-VL-7B fine-tuned with LoRA, GRPO, and post-training) for expert-like multi-modal quality analysis of weather radar nowcasts, trained on the new RQA-70K dataset.
Problem
Score-based metrics for weather radar forecast quality (CSI, SSIM) cannot describe structural/dynamic errors or give interpretable, expert-like assessments, and general MLLMs perform poorly on this analysis.
Approach
A four-task paradigm (frame/sequence rating and assessment) backed by a physics-informed attribute library, hybrid human-plus-script annotation, attribute-informed GPT-4o report generation, and a three-stage SFT, GRPO, post-training pipeline with format and accuracy rewards.
Key results
- On frame rating, RadarQA reaches 61.51% overall accuracy versus 52.67% for o1 and 20.10% for Qwen2.5-VL-7B.
- On sequence rating, RadarQA reaches 66.17% overall accuracy versus 36.70% for the best open-source baseline (InternVL2.5-8B).
- On sequence assessment, RadarQA gets a GPT4Score of 6.58 versus 5.49 for o1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train RadarQA with the multi-stage pipeline (SFT, reinforcement learning, and post-training) rather than the strongest reported baseline | Qwen-2.5-VL-7B · Frame Rating | accuracy (Overall): 52.67 -> 61.51 | Helped |
| Train RadarQA with the multi-stage pipeline (SFT, reinforcement learning, and post-training) rather than the strongest reported baseline | Qwen-2.5-VL-7B · Frame Assessment | GPT4Score: 5.89 -> 6.87 | Helped |
| Train RadarQA with the multi-stage pipeline (SFT, reinforcement learning, and post-training) rather than the strongest reported baseline | Qwen-2.5-VL-7B · Sequence Rating | accuracy (Overall): 36.70 -> 66.17 | Helped |
| Train RadarQA with the multi-stage pipeline (SFT, reinforcement learning, and post-training) rather than the strongest reported baseline | Qwen-2.5-VL-7B · Sequence Assessment | GPT4Score: 5.63 -> 6.58 | Helped |
| Add Stage 1 supervised fine-tuning (ablation) | RadarQA training pipeline · Rating tasks | average accuracy: +40% relative improvement | Helped |
| Add Stage 1 supervised fine-tuning (ablation) | RadarQA training pipeline · Assessment tasks | GPT-4 Score: around 2.5-point increase | Helped |
| Add Stage 2 reinforcement learning to Stage 1 (ablation) | RadarQA training pipeline · Rating and assessment tasks | task performance | Helped |
| Add Stage 3 post-training to Stage 1 (ablation) | RadarQA training pipeline · Rating and assessment tasks | task performance | Helped |
| Use the full three-stage training pipeline rather than partial stage combinations (ablation) | RadarQA training pipeline · Frame Rating, Frame Assessment, Sequence Rating, and Sequence Assessment | task performance | Helped |
| Jointly train across multiple tasks rather than training a separate single-task model (ablation) | RadarQA · Each task's respective evaluation setting | task-specific metric | Helped |
Surprising findings
- InternVL2.5-8B outperforms the larger Qwen2.5-VL-72B on sequence rating (36.70% vs 19.80% overall).
- API-based models score only about 20-30% accuracy on sequence rating overall.
Limitations
- Task paradigm not fully unified; does not support comparison between two predictions.
- Fine-grained descriptions remain unsatisfactory.
- Whether assessment outputs can serve as feedback or rewards for forecasting models is unexplored.
Experimental setup
Datasets: SEVIR RQA-70K RawRQA-20K
Benchmarks: RQA-70K validation split out-of-distribution radar reconstruction data (DiffSR, SRViT, U-Net outputs)
Metrics: accuracy BLEU BERTScore ROUGE_L METEOR GPT4Score
Baselines: Qwen2.5-VL-7B Qwen2.5-VL-72B InternVL2.5-8B GPT-4o Claude 3.7 Sonnet Gemini 2.5 Pro o1
Builds on: GRPO Q-Insight LoRA Qwen2.5-VL
Closest papers by meaning
- WeatherSyn: An Instruction Tuning MLLM For Weather Forecasting Report Generation (ICML 2026 · similarity 0.87)
- Integrating Multi-Source Data for Long Sequence Precipitation Forecasting (AAAI 2025 · similarity 0.85)
- AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science (NeurIPS 2025 · similarity 0.83)
- CyPortQA: Benchmarking Multimodal Large Language Models for Cyclone Preparedness in Port Operation (AAAI 2026 · similarity 0.83)
- TSQAgent: Rating Time Series Data Quality via Dedicated Agentic Reasoning (NeurIPS 2026 · similarity 0.83)
- Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.