Research map / Multi-agent systems and games / LLM agents
Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DORA, a benchmark of 515 expert-authored end-to-end disaster-response tasks over 45 real events with a 108-tool MCP library and replayable gold trajectories, evaluating 13 frontier LLM agents.
Problem
Existing agent and remote-sensing benchmarks isolate perception or generic tool use, leaving end-to-end emergency operations workflows (multi-sensor fusion, spatial reasoning, evacuation planning, reporting) untested.
Approach
A disaster-specific agentic benchmark with expert-written queries, symbolic gold tool trajectories replayed deterministically, a heterogeneous geospatial data manifest, and an MCP tool library, scored with typed final-answer operators plus trajectory metrics.
Key results
- Best model Gemini-3.0-Flash reaches 53.74% average final-answer accuracy, about 26 points below the 80.48% gold trajectory.
- Gold tool-order hints improve final-answer accuracy by only 1.08–4.40%, while the best scaffold (Plan-then-Execute on Gemma-4-31B) gives +3.24%.
- Agent-to-gold gap widens from about 7% on short 2–3 step tasks to 56% on tasks with 11 or more steps.
Surprising findings
- Gemma-4-31B outperforms the 120B GPT-OSS model, suggesting agentic post-training quality matters more than parameter scale.
- Tool-free VLMs score about 18% and the most efficient model (GPT-OSS-120B) has low accuracy, showing confident selection of wrong tools.
Limitations
- Gold trajectory uses model-backed perception tools rather than GT masks, so the oracle is not perfect.
- Trajectory metrics are diagnostic only since multiple valid trajectories exist.
- Perception tools are trained on public data and may have limited accuracy; LLM-as-judge and human scoring only in appendix.
Experimental setup
Datasets: xBD DisasterM3 BRIGHT NAIP Maxar Open Data Landslide4Sense GVLM-CD Planet imagery RescueNet CRASAR-U-DRoIDS OpenEarthMap OpenStreetMap
Benchmarks: DORA
Metrics: final-answer accuracy Tool-Any-Order Tool-In-Order Tool-Exact-Match Parameter Accuracy Efficiency
Baselines: Gemini-3.0-Flash Qwen3-VL-235B GPT-5.4 Claude-Sonnet-4.6 Grok-4.1-Fast Qwen3.5-397B-A17B Gemma-4-31B DeepSeek-V3.2 GPT-OSS-120B Plan-then-Execute Reflexion ReWOO
Builds on: ReAct Plan-then-Execute Reflexion ReWOO xBD
Closest papers by meaning
- DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response (NeurIPS 2025 · similarity 0.87)
- Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agents (ICLR 2026 · similarity 0.86)
- MONITRS: Multimodal Observations of Natural Incidents Through Remote Sensing (NeurIPS 2025 · similarity 0.85)
- GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing (CVPR 2026 · similarity 0.84)
- Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challenges (NeurIPS 2025 · similarity 0.83)
- Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models (ICLR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.