Research map / Reinforcement learning / Policy gradient and actor-critic
Feasible Reachable Policy Iteration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes feasible reachable (FR) functions and feasible reachable policy iteration (FRPI), which prunes the policy search space to states that can safely reach a goal, and integrates it with SAC for safe goal-reaching RL.
Problem
Safe goal-reaching RL suffers from exploration termination due to safety constraints and sparse goal rewards; existing feasible-region methods are slow to converge and keep many feasible states that cannot reach the goal in finite time.
Approach
Defines a feasible reachable function that combines constraint feasibility and goal reachability in finite time, shows it satisfies a self-consistency condition and a risky Bellman equation solvable by fixed-point iteration, and uses region expansion with region-wise policy improvement to shrink the search space to the maximal FR region.
Key results
- On Safety Gym tasks, FRPI-SAC achieves near-zero constraint violations with lower and more stable episode cost than SAC-Lagrangian and FPI-SAC, and returns comparable to SAC.
- FRPI identifies the feasible reachable region at about 5k iterations in ACC, whereas FPI only shows a feasible region around 75k iterations.
- The conclusion claims convergence to the maximum feasible reachable region more than three to five times faster than other safe RL approaches on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FRPI instead of FPI for feasible-region identification | ACC · ACC | iterations until feasible region appears: 75k -> 5k iterations | Helped |
| Use FRPI instead of FPI for feasible-region identification | Quadrotor · Quadrotor trajectory tracking | iterations until feasible region appears: 150k -> 5k iterations | Helped |
| Use FRPI instead of FPI | FRPI-SAC · ACC | region expansion speed | Helped |
| Use FRPI instead of FPI | FRPI-SAC · Quadrotor trajectory tracking | region expansion speed | Helped |
| Use FRPI-SAC instead of SAC-Lagrangian and FPI-SAC | FRPI-SAC · Safety Gym PointGoal, PointPush, CarGoal, and CarPush | episode cost | Helped |
| Use FRPI-SAC instead of other compared algorithms | FRPI-SAC · Safety Gym PointGoal, PointPush, CarGoal, and CarPush | episode return | Helped |
| Use FRPI-SAC instead of SAC | FRPI-SAC · Safety Gym tasks | performance with limited data | Helped |
| Prune policy space using FRPI | FRPI · Frozen Lake | policy-space pruning efficiency | Helped |
Surprising findings
- Unconstrained SAC reaches similar returns on some tasks but at the cost of safety violations.
Limitations
- Requires manually designed goal function for regulating tasks such as ACC.
- Performance depends on the effectiveness of goal function design.
- Theory assumes deterministic MDP.
Experimental setup
Datasets: Frozen Lake Adaptive Cruise Control Quadrotor (safe-control-gym) Safety Gym PointGoal, PointPush, CarGoal, CarPush
Benchmarks: Frozen Lake Adaptive Cruise Control Quadrotor trajectory tracking Safety Gym PointGoal Safety Gym PointPush Safety Gym CarGoal Safety Gym CarPush
Metrics: episode return episode cost sample efficiency FR region size
Baselines: SAC-Lagrangian FPI-SAC SAC penalty function Q-learning
Builds on: Soft Actor-Critic Feasible region / safety index methods (FPI) Lagrangian safe RL
Closest papers by meaning
- Iterative Reachability Estimation for Safe Reinforcement Learning (NeurIPS 2023 · similarity 0.88)
- Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learning (ICLR 2026 · similarity 0.87)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.86)
- Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction (NeurIPS 2026 · similarity 0.85)
- Provably Efficient RL for Linear MDPs under Instantaneous Safety Constraints in Non-Convex Feature Spaces (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.