Research map / Multi-agent systems and games / Human-AI and goal-conditioned
HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HAZARD, a simulated embodied benchmark on ThreeDWorld with fire, flood, and wind disaster scenarios where agents must rescue objects, and evaluates RL, rule, search, and LLM-based agents.
Problem
Existing embodied simulators lack environment-driven, unexpected dynamic changes, so agents' decision-making under changing environments (disasters) is under-assessed.
Approach
Adds physics and visual simulation of fire (temperature and spreading model), flood (buoyancy and drag), and wind (aerodynamic force with turbulence) to ThreeDWorld, framed as an object-rescue task with value/step/damage metrics and an LLM API that compresses low-level navigation via A* into high-level actions.
Key results
- GPT-4 LLM agent achieves 77.8 value on fire (without perception) vs 75.9 for MCTS and 46.1 for RL.
- On wind, GPT-4 LLM agent reaches 31.1 value vs 18.0 for MCTS and 12.4 for RL, while all methods remain low.
- With perception, performance drops across methods; e.g., GPT-4 flood value 38.2 vs 45.7 without perception.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of MCTS | HAZARD agents · Fire, without perception | Value: 75.9 -> 77.8 | Helped |
| Use GPT-4 instead of MCTS | HAZARD agents · Flood, without perception | Value: 43.7 -> 45.7 | Helped |
| Use GPT-4 instead of MCTS | HAZARD agents · Wind, without perception | Value: 18.0 -> 31.1 | Helped |
| Use GPT-4 instead of MCTS | HAZARD agents · Fire, with perception | Value: 59.2 -> 67.7 | Helped |
| Use GPT-4 instead of RL | HAZARD agents · Flood, with perception | Value: 33.1 -> 38.2 | Helped |
| Use GPT-4 instead of MCTS | HAZARD agents · Wind, with perception | Value: 18.0 -> 33.9 | Helped |
| Use GPT-4 instead of MCTS | HAZARD agents · Fire, without perception | Step: 150.1 -> 159.9 | Hurt |
| Use GPT-4 instead of MCTS | HAZARD agents · Flood, without perception | Step: 146.6 -> 142.9 | Helped |
| Use GPT-4 instead of Greedy | HAZARD agents · Wind, without perception | Step: 444.0 -> 590.1 | Hurt |
| Use perception model instead of semantic input | HAZARD agents · Fire | Value: 77.8 -> 67.7 | Hurt |
| Use perception model instead of semantic input | HAZARD agents · Flood | Value: 45.7 -> 38.2 | Hurt |
| Use perception model instead of semantic input | HAZARD agents · Wind | Value: 31.1 -> 33.9 | Helped |
Surprising findings
- Wind scenario is extremely hard: Rule and Greedy agents obtain near-zero value, and even the best method reaches only about 31-34 value.
- Perception difficulty can help in wind scenarios, since agents that fail to see blown-away objects stop chasing them.
Limitations
- Challenge focuses only on object rescue; agents cannot mitigate hazards (e.g., extinguish fire).
- Hazard effects on agents are disabled by default; the agent-affected setting is only partially explored and fire-affected agents are not evaluated.
- Physics for wind and flood uses hand-adjusted, idealized force models.
Experimental setup
Datasets: HAZARD procedurally generated scenes
Benchmarks: HAZARD fire HAZARD flood HAZARD wind
Metrics: rescued value rate averaged rescue step averaged damage rate
Baselines: Random agent Rule-based agent Greedy agent RL (PPO) MCTS agent LLM (Llama-13b) LLM (GPT-3.5)
Builds on: ThreeDWorld ProcGenKitchen PPO Monte Carlo Tree Search
Closest papers by meaning
- AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions (CVPR 2026 · similarity 0.89)
- Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (NeurIPS 2024 · similarity 0.89)
- IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks (AAAI 2026 · similarity 0.88)
- From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents (ICLR 2026 · similarity 0.87)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.86)
- VirtualEnv: A Platform for Embodied AI Research (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.