Research map / Reinforcement learning / Visual and representation RL
HYPRL: Reinforcement Learning of Control Policies for Hyperproperties
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HYPRL, which Skolemizes HyperLTL formulas and uses quantitative robustness-based reward shaping so off-the-shelf RL (DQN, PPO, CQ-learning) learns multi-agent policies that maximize satisfaction of quantifier-alternating hyperproperties.
Problem
Reward shaping for multi-agent RL tasks with inter-agent dependencies, where existing approaches implicitly treat specifications as universally quantified and miss optimal joint policies.
Approach
Skolemize HyperLTL formulas with quantifier alternation so existentially quantified agent traces become Skolem functions of universal traces, then define quantitative (robustness) semantics over zipped traces to produce per-step rewards for standard RL algorithms.
Key results
- In the 3-agent SRL setting, HYPRL with CQ-learning reached goals in ~74 steps on ISR versus shielding failing to reach goals within the 100-step bound.
- On Deep Sea Treasure, PPO+HYPRL collected 22.93 treasures vs 4.31 for PPO at 500 episodes.
- In the wildfire 5^2 setting, PPO+HYPRL rescued victims within about 8057 steps while PPO exceeded the step bound.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HypRL to PPO | PPO · DST, 500 and 1000 episodes | total treasures: 4.31$\pm1.2$ -> 22.93$\pm2.2$; 3.22$\pm0.8$ -> 23.97$\pm2.2$ | Helped |
| Add HypRL to DQN | DQN · DST, 500 and 1000 episodes | total treasures: 1.08$\pm0.2$ -> 4.12$\pm1.4$; 1.45$\pm0.2$ -> 4.43$\pm0.8$ | Helped |
| Add HypRL to PPO | PPO · Wildfire 3² | Steps O2: 787.03$\pm31.8$ -> 143.550$\pm1.1$ | Helped |
| Add HypRL to PPO | PPO · Wildfire 5² and 8² | victim rescue within step bound: S/B -> 8057.5$\pm121.4$; S/B -> 386.2$\pm$80.5 | Helped |
| Add HypRL to PPO | PPO · Wildfire 10² | fire extinguishing within step bound: S/B -> 21272.8$\pm$3579.0 | Helped |
| Add HypRL to CQ-learning | CQ-learning · SRL, 2 agents across ISR, Pentagon, SUNY, and MIT | steps: ISR: 27.95$\pm7.4$ -> 7.58$\pm0.3$; Pentagon: 36.46$\pm7.7$ -> 11.90$\pm4.6$; SUNY: 11.99$\pm0.5$ -> 12.48$\pm0.6$; MIT: | Mixed |
Surprising findings
- Shield synthesis avoids all collisions but at a large cost in steps to reach goals, failing to reach goals within 100 steps in several 3-agent maps.
Limitations
- Does not support fully decentralized control policies with local observations and rewards.
- Assumes fully observable MDP; does not handle POMDPs.
- Skolemization and robustness computation for sampled traces is limited to finite-trace semantics with fixed path lengths.
Experimental setup
Datasets: Safe RL grid-world maps (ISR, Pentagon, SUNY, MIT) Deep Sea Treasure Post Correspondence Problem instances Wildfire grid-worlds (3^2 to 10^2)
Benchmarks: Safe RL grid worlds (ISR, Pentagon, SUNY, MIT) Deep Sea Treasure Post Correspondence Problem Wildfire scenario
Metrics: steps to goal collisions treasures collected successful matches steps to rescue victims and extinguish fires distance between agents
Baselines: CQ-learning CQ-learning + shield synthesis DQN with manual reward PPO with manual reward
Builds on: HyperLTL DQN PPO CQ-learning SPECTRL
Closest papers by meaning
- HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observation (ICML 2026 · similarity 0.91)
- A Composable Specification Language for Reinforcement Learning Tasks (NeurIPS 2019 · similarity 0.88)
- Automating the Refinement of Reinforcement Learning Specifications (ICLR 2026 · similarity 0.88)
- Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.87)
- On the Expressivity of Objective-Specification Formalisms in Reinforcement Learning (ICLR 2024 · similarity 0.87)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.