Research map / Vision-language and multimodal / Multimodal LLMs

CyPortQA: Benchmarking Multimodal Large Language Models for Cyclone Preparedness in Port Operation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CyPortQA, a multimodal QA benchmark (117,178 pairs from 2,917 cyclone-port scenarios) for evaluating MLLMs on port cyclone preparedness across situation understanding, impact estimation, and decision reasoning.

Problem

MLLMs' accuracy and reliability for port cyclone preparedness, which requires synthesizing NOAA forecast products, port impact records, and USCG port-condition bulletins, has not been rigorously evaluated.

Approach

Builds scenario-level structured JSON records fusing NOAA tropical cyclone products, AIS-based port impact data, and USCG bulletins, then auto-generates templated QA pairs across three preparedness stages, with over/under-reaction metrics for decision reasoning.

Key results

• ChatGPT-4o achieves the highest accuracy on situation understanding, with True/False accuracy of 0.81-0.83 on spatial and temporal items.
• Gemini2.5 leads impact estimation, with S2.1 time-of-impact score 0.69 and S2.2 recovery duration score 0.72 versus ChatGPT-4o's 0.55 and 0.70.
• Descriptive uncertainty questions (S1.4 SA-Desc) score very low, e.g., 0.16 for ChatGPT-4o and 0.01 for LLaVA-1.6, indicating difficulty converting inputs to probabilistic risk statements.

Surprising findings

• Most models tend to under-react, assigning less stringent port condition bulletins than required, especially Gemini-2.5, LLaMA-3.2, and Qwen-2.5-VL.
• Qwen2.5-VL outperforms ChatGPT-4o on S2.3 multiple-choice impact severity (0.46 vs 0.44) despite weaker overall understanding.

Limitations

• Benchmark built only from U.S. tropical cyclone cases, limiting generalization to other regions.
• Smaller ports may have less geographic knowledge in MLLMs, degrading spatial awareness.
• Reliance on ChatGPT o3 for scenario JSON encoding with only 10% expert spot-check.

Experimental setup

Datasets: NOAA tropical cyclone products CyPort port operational performance dataset USCG port condition bulletins

Benchmarks: CyPortQA

Metrics: accuracy tolerance-based accuracy LLM judge score under-reaction rate over-reaction rate

Baselines: LLaVA-1.6 Llama-3.2-Vision Gemma-3 Qwen2.5-VL Mistral-Small-3.1 ChatGPT-4o Gemini-2.5-Flash-Lite

Builds on: LLM-as-a-judge

Closest papers by meaning

• Early Detection of Extreme Storm Tide Events Using Multimodal Data Processing (AAAI 2024 · similarity 0.83)
• RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts (NeurIPS 2025 · similarity 0.83)
• Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations (NeurIPS 2026 · similarity 0.83)
• TC-Diffuser: Bi-Condition Multi-Modal Diffusion for Tropical Cyclone Forecasting (AAAI 2025 · similarity 0.82)
• Off to new Shores: A Dataset & Benchmark for (near-)coastal Flood Inundation Forecasting (NeurIPS 2024 · similarity 0.81)
• ENC-Bench: A Benchmark for Evaluating Multimodal Large Language Models in Electronic Navigational Chart Understanding (CVPR 2026 · similarity 0.81)