Research map / Reinforcement learning / Visual and representation RL
Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoSHRL, a hierarchical RL method pairing a goal-conditioned distributional RL low-level agent with a ConstrainedRRT* upper-level search that enforces expected or CVaR cost constraints over long-horizon navigation without retraining.
Problem
Long-horizon goal-directed RL with trajectory-level safety constraints, including percentile (CVaR) cost constraints, where existing constrained RL methods only handle short horizons and need retraining when thresholds change.
Approach
Combine an upper-level Informed RRT*-based search that builds waypoints under CVaR cost constraints (checked by convolving segment cost distributions) with a lower-level goal-conditioned distributional RL agent that estimates local reward and cost distributions, so cost thresholds can change without retraining.
Key results
- CoSHRL achieves about 100% success rate in the obstacle-only point maze, outperforming SORB with a growing margin as difficulty increases.
- In the static-hazard maze, CoSHRL reaches the goal with nearly 100% success while SAC-Lagrangian and WCSAC achieve near 0% success on this long-horizon task; GRL succeeds under 20%.
- In Safe-ViZDoom, CoSHRL achieves over 95% success with cost-limit violations of only 2%, 4%, 5% for K=0,4,8, while unconstrained SORB and GRL exceed cost 8 in over half of trajectories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoSHRL with risk level α=0.1 instead of α=0.9 | CoSHRL · Stochastic-cost point maze | trajectories exceeding cost limit: 16% -> 7% | Helped |
| Use CoSHRL with risk level α=0.1 instead of α=0.9 | CoSHRL · Stochastic-cost point maze | expected negated reward (ENR): 47.83 -> 48.60 | Helped |
| Use CoSHRL with risk level α=0.1 instead of α=0.9 | CoSHRL · Stochastic-cost point maze | expected cost (EC): 7.90 -> 6.47 | Helped |
| Use CoSHRL with risk level α=0.1 instead of α=0.9 | CoSHRL · Stochastic-cost point maze | cost-CVaR-0.1: 14.67 -> 12.11 | Hurt |
| Use CoSHRL with risk level α=0.9 instead of SORB | Stochastic-cost point maze · Stochastic-cost point maze | expected negated reward (ENR): 52.98 -> 47.83 | Helped |
| Use CoSHRL with risk level α=0.9 instead of SORB | Stochastic-cost point maze · Stochastic-cost point maze | cost-CVaR-0.9: 8.97 -> 8.47 | Helped |
Surprising findings
- WCSAC and SAC-Lagrangian achieve roughly 0% success on the long-horizon hazard maze task.
- CoSHRL slightly violates the CVaR bound at the tight level alpha=0.1 due to discretization and truncation in distributional RL.
Limitations
- Assumes the low-level agent reliably reaches each waypoint; poor generalization can declare reachable states unreachable, yielding no path (success rate not exactly 100%).
- Cost estimates degrade as paths traverse more hazardous area, and CVaR constraint is slightly violated at tight alpha.
- Asymptotic optimality proof relies on robustly optimal solution assumptions and a theoretical framework from prior RRT* analysis.
Experimental setup
Datasets: point maze environments Safe-ViZDoom
Benchmarks: 2D point maze navigation with obstacles 2D point maze with hazards (static and stochastic cost) Safe-ViZDoom image navigation
Metrics: success rate negated reward (path length) cost violation proportion CVaR of cost expected cost
Baselines: SAC-Lagrangian WCSAC SORB Goal-conditioned RL (GRL)
Builds on: Informed RRT* RRT* SORB goal-conditioned RL distributional RL
Closest papers by meaning
- Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Constrained Reinforcement Learning in Hard Exploration Problems (AAAI 2023 · similarity 0.91)
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.90)
- Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction (NeurIPS 2026 · similarity 0.89)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.