Research map / Reinforcement learning / Policy gradient and actor-critic
Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MICE, which adds a memory-driven intrinsic cost (pseudo-count of visits to stored unsafe states) and bias correction to the cost critic to counter cost underestimation, reducing constraint violations in CPO and PID Lagrangian style constrained RL.
Problem
Constrained RL methods often violate constraints during training because the cost value function is underestimated, so high-cost states look deceptively safe.
Approach
A flashbulb memory of unsafe states, randomly projected to a lower-dimensional space, yields a kNN-based pseudo-count intrinsic cost added to the cost target, with an adaptive factor beta corrected by the estimation bias; the method is wrapped in CPO and PID Lagrangian optimizers with a trust region.
Key results
- MICE keeps constraint violations near zero on Safety MuJoCo while matching or exceeding baseline return, per the paper's learning curves.
- Cost value estimates in MICE are reported to be significantly higher than baselines and their bias approaches zero over training.
- Sensitivity runs on SafetyPointGoal1 with thresholds 0, 15 and 25 show the method adapts its constraint satisfaction to the threshold.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add memory-driven intrinsic cost and bias correction | MICE · Safety Gym | constraint violations | Helped |
| Add memory-driven intrinsic cost and bias correction | MICE · Safety Gym | policy performance | Mixed |
| Add memory-driven intrinsic cost and bias correction | MICE · Safety MuJoCo | constraint violations | Helped |
| Add memory-driven intrinsic cost and bias correction | MICE · Safety MuJoCo | policy performance | Helped |
| Replace memory-driven intrinsic cost with a fixed constant (ablation) | MICE · Safety Gym and Safety MuJoCo | policy performance | Hurt |
| Use fixed cost constant 5 instead of fixed cost constant 3 (ablation) | MICE · Safety Gym and Safety MuJoCo | constraint violations | Hurt |
| Add the random projection layer (ablation) | MICE · Safety Gym and Safety MuJoCo | computational complexity | Helped |
| Add the random projection layer (ablation) | MICE · Safety Gym and Safety MuJoCo | policy performance | No clear effect |
| Increase the number of nearest neighbors k (ablation) | MICE · multiple environments | policy safety | Helped |
| Increase the number of nearest neighbors k (ablation) | MICE · multiple environments | computational overhead | Hurt |
| Use a smaller number of nearest neighbors k (ablation) | MICE · multiple environments | constraint violations | Hurt |
| Set the constraint threshold to 0 (ablation) | MICE · SafetyPointGoal1-v0 | constraint satisfaction | Helped |
Surprising findings
- Adding larger fixed constants to the cost estimate (e.g., 5) produced more constraint violations than constant 3, so conservatism alone does not reduce violations.
Limitations
- Relies on extrinsic costs being noise-free to decide memory inclusion; noisy-cost settings are only proposed as future work.
- Scalability to offline memory construction (e.g., autonomous driving) is discussed but not evaluated.
- Theoretical bounds rely on assumptions borrowed from TD3 and Double Q-learning.
Experimental setup
Datasets: Safety Gym Safety MuJoCo
Benchmarks: Safety Gym Safety MuJoCo
Metrics: cumulative discounted reward cumulative discounted cost cost value estimation bias
Baselines: PID Lagrangian CUP Saute Simmer CPO
Builds on: CPO PID Lagrangian NGU TD3 CUP
Closest papers by meaning
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.89)
- Extreme Value Policy Optimization for Safe Reinforcement Learning (ICML 2025 · similarity 0.88)
- Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.87)
- Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation (ICML 2024 · similarity 0.87)
- Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.