Research map / Reinforcement learning / Policy gradient and actor-critic
Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RL-AR, which combines an MPC safety regularizer with an off-policy SAC agent via a state-dependent focus module, ensuring safety during training in single-life critical control tasks while matching model-free RL returns.
Problem
Safe reinforcement learning for critical single-life control systems, where no failures are tolerated during training and the estimated environment model may be inaccurate.
Approach
Combines a constrained-MPC safety regularizer with an off-policy SAC agent through a learned state-dependent weight beta(s), initialized to favor the safe MPC policy and gradually shifting to the RL policy for well-explored states; theoretically shown to bound deviation from the regularizer and to converge to the RL optimum.
Key results
- RL-AR had zero failed training episodes out of the first 100 in the actual environment, while SAC averaged the most failures across environments.
- After convergence, RL-AR outperformed SAC in normalized return in Glucose, BiGlucose, and CSTR, and was competitive with SAC in Cart Pole.
- RL-AR remained safe with parameter mismatches of more than 60%; failures only occurred when p2 <= 3/8 of estimated value and n <= 6/16 of estimated value in Glucose.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine constrained MPC and SAC actions using a learned state-dependent focus module | RL-AR · Glucose, BiGlucose, CSTR, Cart Pole | training failures | Helped |
| Combine constrained MPC and SAC actions using a learned state-dependent focus module | RL-AR · Glucose, BiGlucose, CSTR, Cart Pole | normalized return during training | Helped |
| Use adaptive regularization with the safety regularizer and SAC | RL-AR · Glucose, BiGlucose, CSTR | converged normalized return | Helped |
| Use adaptive regularization with the safety regularizer and SAC | RL-AR · Cart Pole | converged normalized return | No clear effect |
| Pretrain SAC, CPO, and SEditor on the estimated model | SAC, CPO, and SEditor · Actual environments during training | training failures | Mixed |
| Increase discrepancy between the estimated and actual Glucose model parameters | RL-AR · Glucose | training failures | Hurt |
Surprising findings
- RL-AR's return outperformed SAC after convergence in three of four environments, despite SAC being the RL agent it builds on.
Limitations
- Assumes the estimated model has reasonable accuracy to derive a viable safety regularizer.
- Safety during training depends on the quality of the estimated model; large discrepancies cause failures.
- Early in training RL-AR return can fall below the static MPC policy in CSTR and Cart Pole.
Experimental setup
Datasets: simulated Glucose environment simulated BiGlucose environment simulated CSTR environment simulated Cart Pole environment
Benchmarks: Glucose BiGlucose CSTR Cart Pole
Metrics: normalized episodic return number of failed episodes out of first 100 step time
Baselines: MPC SAC Residual Policy Learning CPO SEditor
Builds on: Soft Actor-Critic Residual Policy Learning Model Predictive Control
Closest papers by meaning
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.89)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.89)
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.88)
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.88)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.