Research map / Reinforcement learning / Visual and representation RL
Optimizing Heat Alert Issuance with Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds BROACH, a Bayesian-reward RL environment for heat alert issuance from Medicare hospitalizations and NWS alerts, and shows standard RL (DQN, QR-DQN, TRPO, A2C) only beats the NWS policy when alerts are restricted to very hot days.
Problem
Optimizing when to issue heat alerts under a fixed alert budget to reduce heat-related hospitalizations, where health effects are low-signal, spatially heterogeneous, and confounded in observational data.
Approach
A county-level RL environment whose reward comes from a variational Bayesian hierarchical hospitalization model with data-driven priors, and whose transitions replay real observed weather trajectories augmented with weather from same-climate-region counties; RL policies are restricted to days above a per-county quantile heat index threshold.
Key results
- A2C.QHI achieved the largest median return gain over NWS (0.042, WMW p=0.011), while unrestricted DQN, QR-DQN, TRPO and A2C all performed worse than NWS (median differences -0.063 to -0.123).
- Extrapolated nationally, A2C.QHI implemented across all US counties was estimated to reduce about 222 NOHR hospitalizations per year (95% CI -491 to 1131).
- Deterministic evaluation of TRPO and A2C policies performed poorly, showing that stochastic policies are needed under this rewards model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Restrict alerts to days above a county-specific QHI threshold (ablation) | DQN, QR-DQN, TRPO, and A2C · Evaluation years 2007, 2011, 2015 | Median difference in average return vs NWS: DQN: -0.123 -> 0.03; QR-DQN: -0.117 -> 0.035; TRPO: -0.065 -> 0.038; A2C: -0.063 -> 0.042 | Helped |
| Add perfect future information to the state (ablation) | DQN and QR-DQN · Evaluation years 2007, 2011, 2015 | Median difference in average return vs NWS: DQN: -0.123 -> -0.417; QR-DQN: -0.117 -> -0.42 | Hurt |
| Add perfect future information to the state (ablation) | TRPO with QHI restriction · Evaluation years 2007, 2011, 2015 | Median difference in average return vs NWS: 0.038 -> 0.046 | Helped |
| Add perfect future information to the state (ablation) | A2C with QHI restriction · Evaluation years 2007, 2011, 2015 | Median difference in average return vs NWS: 0.042 -> 0.036 | Hurt |
| Evaluate the stochastic policy deterministically (ablation) | TRPO and A2C with QHI restriction · Evaluation years 2007, 2011, 2015 | Median difference in average return vs NWS: TRPO: 0.038 -> -0.662; A2C: 0.042 -> -0.886 | Hurt |
| Use 10 rather than 20 quantiles (ablation) | QR-DQN · Heat-alert policy optimization | Average return | Helped |
| Use the QHI-restricted A2C policy instead of NWS | A2C · Evaluation years 2007, 2011, 2015 | Median difference in average return: 0.042 | Helped |
| Use random alert-day selection instead of NWS | Random baseline · Evaluation years 2007, 2011, 2015 | Median difference in average return: -0.015 | No clear effect |
| Use the basic NWS heat-index guidelines instead of NWS | basic.nws · Evaluation years 2007, 2011, 2015 | Median difference in average return: -0.286 | Hurt |
| Issue alerts on the b hottest days of the summer | topk oracle baseline · Evaluation years 2007, 2011, 2015 | Median difference in average return: 0.022 | Helped |
| Issue alerts above an optimized county-specific QHI threshold until budget runs out | aa.qhi baseline · Evaluation years 2007, 2011, 2015 | Median difference in average return: 0.03 | Helped |
Surprising findings
- Restricting RL to very hot days is necessary; a simple always-alert-above-QHI-threshold policy (aa.qhi) performs nearly as well as the best RL policy.
- Including perfect future weather information made DQN and QR-DQN noticeably worse, while only TRPO benefited.
Limitations
- Fixed alert budget equal to observed NWS counts; does not model alert fatigue beyond the rewards model or monetary budgets.
- Rewards model assumes effects are estimated from observational Medicare data with unobserved mediation; low R^2 (0.102) for rate prediction.
- Evaluation on only 30 counties and 3 held-out years; future-information results assume perfect forecasts.
Experimental setup
Datasets: NWS heat alerts (2006-2016) heat index weather data Medicare Part A hospitalization claims county covariates (population density, median household income, broadband, election returns, PM2.5)
Benchmarks: BROACH heat alert environment (30 counties, evaluation years 2007, 2011, 2015)
Metrics: average return per episode median difference in return vs NWS Wilcoxon-Mann-Whitney statistic and p-value
Baselines: NWS policy random basic.nws aa.qhi topk DQN QR-DQN TRPO A2C
Builds on: Stable-Baselines3 Pyro Gymnasium Deep Q-learning TRPO
Closest papers by meaning
- Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach (NeurIPS 2025 · similarity 0.84)
- InvestESG: A multi-agent reinforcement learning benchmark for studying climate investment as a social dilemma (ICLR 2025 · similarity 0.82)
- Learning to Reuse Policies in State Evolvable Environments (ICML 2025 · similarity 0.82)
- Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits (AAAI 2025 · similarity 0.82)
- AI for Global Climate Cooperation: Modeling Global Climate Negotiations, Agreements, and Long-Term Cooperation in RICE-N (ICML 2025 · similarity 0.82)
- When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.