Research map / Reinforcement learning / Visual and representation RL
Solving Minimum-Cost Reach Avoid using Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RC-PPO, a two-phase PPO-based method that solves minimum-cost reach-avoid problems by lifting to augmented dynamics (safety flag and cost budget z) and optimizing the least feasible cost bound z.
Problem
Minimum-cost reach-avoid control: reach a goal, avoid unsafe sets, and minimize cumulative cost with flexible final time; standard RL uses weighted-sum surrogates or CMDPs that yield suboptimal policies.
Approach
Augment the state with a safety flag and a remaining cost budget z so that the minimum-cost reach-avoid problem becomes a reach problem on augmented dynamics; learn a z-conditioned PPO policy with a discounted reach Bellman operator, then find the least feasible z by bisection or a regressed map.
Key results
- RC-PPO achieves up to 57% lower cumulative cost than existing methods on minimum-cost reach-avoid benchmarks in MuJoCo, with comparable goal-reaching rates.
- RC-PPO outperforms the entire Pareto front of PPO across grid-searched reward coefficients in the surrogate CMDP formulation.
- RC-PPO achieves the highest reach rates in all benchmark tasks under the sparse reward setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RC-PPO instead of existing methods | minimum-cost reach-avoid benchmarks on Mujoco · Mujoco benchmark suite | cumulative cost: up to 57% lower | Helped |
| Use RC-PPO instead of baseline methods | minimum-cost reach-avoid benchmarks · Mujoco benchmark suite | reach rate | No clear effect |
| Use RC-PPO instead of baseline algorithms | sparse-reward reach-avoid tasks · Pendulum, Safety Hopper, WindField, PointGoal, Safety HalfCheetah, FixedWing | reach rate | Helped |
| Use RC-PPO instead of baseline algorithms | reward-shaped reach-avoid tasks · Pendulum, PointGoal, Safety HalfCheetah, WindField | cumulative cost | Helped |
| Use RC-PPO instead of the Pareto front from the surrogate CMDP grid search | PPO with varied reward coefficients · minimum-cost reach-avoid benchmarks | cost and reach-rate trade-off | Helped |
| Add control noise | RC-PPO · minimum-cost reach-avoid benchmarks | cumulative cost | Helped |
Surprising findings
- Theoretical example shows no choice of CMDP hyperparameter yields the optimal solution of the minimum-cost reach-avoid problem.
Limitations
- Theory assumes deterministic dynamics, precluding domain randomization.
- Folding safety into the goal via a binary flag means two unable-to-reach policies can share a value even if one is unsafe.
Experimental setup
Datasets: Pendulum Safety Hopper Safety HalfCheetah PointGoal (Safety Gym) WindField quadrotor FixedWing
Benchmarks: Pendulum Safety Hopper Safety HalfCheetah PointGoal WindField FixedWing
Metrics: reach rate cumulative cost (IQM)
Baselines: CRL CPPO RESPO PPO_beta_L PPO_beta_H SAC_beta_L SAC_beta_H
Builds on: PPO generalized advantage estimation Hamilton-Jacobi reachability RESPO CPPO
Closest papers by meaning
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.86)
- PPO in the Fisher-Rao geometry (NeurIPS 2026 · similarity 0.86)
- Cost-Aware Learning (NeurIPS 2026 · similarity 0.85)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Simple Policy Optimization (ICML 2025 · similarity 0.85)
- P2BPO: Permeable Penalty Barrier-Based Policy Optimization for Safe RL (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.