Research map / Reinforcement learning / Visual and representation RL
Active Reinforcement Learning for Robust Building Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ActivePLR, which uses MC-dropout critic uncertainty and gradient ascent over environment weather offsets to generate training environments for robust RL HVAC control in Sinergym.
Problem
RL building controllers overfit to normal weather and fail under extreme or drifting weather; existing unsupervised environment design methods generalize across broad distributions and can produce unrealistic environments.
Approach
Uses Monte Carlo Dropout uncertainty of the PPO critic and gradient ascent on continuous environment weather-offset parameters (with hard bounds and a soft distance penalty to a base environment) to select new training environments, integrated into PLR's replay buffer (ActivePLR) or used alone (ActiveRL).
Key results
- ActivePLR improves base-environment reward by 9% over vanilla RL and 24% over RBC, and about 3% over vanilla RL averaged across 6 environments.
- ActivePLR shows a 3.1% relative reward drop under the higher-fidelity Sim2Real test versus 8.5% for vanilla RL.
- ActiveRL achieves a median 5% improvement over vanilla RL across 120 US weather patterns and a median 24% improvement over RBC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ActivePLR instead of Vanilla RL | PPO HVAC controller · 5Zone base environment φ0 | reward: 9% | Helped |
| use ActivePLR instead of RBC | 5Zone HVAC control · 5Zone base environment φ0 | reward: 24% | Helped |
| use ActivePLR instead of Vanilla RL | PPO HVAC controller · 5Zone, all six environments | reward: 3% | Helped |
| use ActivePLR instead of Vanilla RL | PPO HVAC controller · 5Zone, six environments | days with ASHRAE thermal comfort violations: 15% relative decrease | Helped |
| use ActivePLR instead of Vanilla RL | PPO HVAC controller · 5Zone, six environments, higher-fidelity simulation | relative reward drop: 8.5% ... 3.1% | Helped |
| use ActiveRL instead of each baseline | PPO HVAC controller · 5Zone, 120 US weather patterns | reward: 24%; 5% | Helped |
| use ActiveRL instead of ActivePLR | PPO HVAC controller · 5Zone, 120 US weather patterns | reward | No clear effect |
| use domain randomization instead of Vanilla RL | PPO HVAC controller · 5Zone, base environment φ0 | reward: 8% | Hurt |
| use domain randomization instead of Vanilla RL | PPO HVAC controller · 5Zone, extreme weather environments | reward | Hurt |
| increase the soft-constraint strength γ (ablation) | ActiveRL · 5Zone base and extreme weather environments | reward | Helped |
| reduce environment-optimization learning rate η (ablation) | ActiveRL · 5Zone, all environments | reward | Helped |
| decrease η below 0.001 (ablation) | ActiveRL · 5Zone, all environments | learning trajectory: 0.001 | No clear effect |
Surprising findings
- Domain randomization and RPLR did not significantly improve over vanilla RL, and DR did about as well as the RBC on extreme environments.
- Contrary to the hypothesis of a realism-robustness tradeoff, higher soft-constraint weight gamma correlated with better performance in both base and extreme environments.
Limitations
- Relies on simulators (Sinergym/EnergyPlus) rather than real buildings.
- Requires continuous environment configuration variables for gradient ascent.
- ActivePLR and ActiveRL perform similarly on most evaluations, and the reason for ActivePLR's better Sim2Real transfer is speculative.
Experimental setup
Datasets: Sinergym 5Zone simulated weather configurations NREL TMY weather data (120 US patterns)
Benchmarks: Sinergym 5Zone base environment phi_0 5 handcrafted extreme weather environments 120 US TMY weather patterns higher-fidelity EnergyPlus Sim2Real test
Metrics: average reward relative reward drop (Sim2Real) ASHRAE thermal comfort violation days
Baselines: PPO vanilla RL domain randomization RPLR rule-based controller random controller ActiveRL
Builds on: Prioritized Level Replay Robust PLR PPO Monte Carlo Dropout Sinergym
Closest papers by meaning
- Improving Regret Approximation for Unsupervised Dynamic Environment Generation (NeurIPS 2025 · similarity 0.88)
- Unsupervised Training Sequence Design: Efficient and Generalizable Agent Training (AAAI 2024 · similarity 0.86)
- Adversarial Environment Design via Regret-Guided Diffusion Models (NeurIPS 2024 · similarity 0.86)
- No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (NeurIPS 2024 · similarity 0.86)
- Marginal Benefit Driven RL Teacher for Unsupervised Environment Design (AAAI 2025 · similarity 0.85)
- Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.