Research map / Reinforcement learning / Policy gradient and actor-critic
ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ActSafe, a model-based RL algorithm that combines optimistic intrinsic exploration with pessimistic safety constraints via a safe-set expansion, with theory for GP dynamics and a scalable RSSM-ensemble variant for visual control.
Problem
Safe exploration in reinforcement learning: learning policies in continuous state-action CMDPs while satisfying safety constraints during learning, which is hard to scale beyond low-dimensional policies like PID controllers.
Approach
Use a well-calibrated probabilistic dynamics model: plan optimistically w.r.t. epistemic uncertainty (intrinsic reward for expanding the safe set) while enforcing pessimism w.r.t. constraint costs over plausible models, with a practical variant using RSSM ensembles and LBSGD constrained planning.
Key results
- On GP-based Pendulum and Cartpole, ActSafe incurs no costs during learning while Opax and the no-pessimism variant violate constraints.
- On Safety-Gym with vision control, ActSafe significantly reduces constraint violation across tasks compared to baselines, though slightly underperforms LAMBDA in performance while incurring much smaller costs.
- On sparse-reward hard exploration tasks, ActSafe substantially outperforms the Greedy baseline, violating the constraint only once in GotoGoal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add model-uncertainty pessimism to exploration (ablation) | ActSafe without pessimism · Pendulum and Cartpole exploration | constraint violations during learning | Helped |
| Add model-uncertainty pessimism to exploration (ablation) | ActSafe without pessimism · Pendulum and Cartpole exploration | objective reward | Hurt |
| Use intrinsic uncertainty rewards instead of greedy sparse extrinsic-reward exploration | Greedy · Sparse-reward Safety-Gym tasks | task performance | Helped |
| Use intrinsic uncertainty rewards instead of greedy sparse extrinsic-reward exploration | Greedy · GotoGoal | constraint violations | Mixed |
| Use uncertainty-aware safe exploration instead of uniform random actions | Uniform · CartpoleSwingupSparse (RWRL) | task performance | Helped |
| Use safe-set expansion with intrinsic uncertainty rewards rather than uncertainty bonuses alone | Optimistic · CartpoleSwingupSparse (RWRL) | task performance | Helped |
| Replace the LBSGD planner (ablation) | ActSafe · Vision-control experiments | constraint violations | Helped |
Surprising findings
- Pessimism trades off performance: ActSafe converges to a lower reward than unsafe baselines on GP experiments.
Limitations
- Theoretical guarantees hold for GP/RKHS dynamics; the practical variant relies on heuristics such as ensemble disagreement and tuned pessimism parameter lambda.
- Requires an initial safe seed policy set and a 200K-step warm-up period of unconstrained data collection for the neural model.
- Pessimism reduces final performance relative to unsafe baselines.
Experimental setup
Datasets: Pendulum Cartpole Safety-Gym RWRL CartpoleSwingupSparse
Benchmarks: PendulumSwingup CartpoleSwingup Safety-Gym RWRL CartpoleSwingupSparse
Metrics: cumulative cost constraint violation objective performance episodic return
Baselines: LAMBDA BSRP-Lag CPO Opax Greedy Uniform Optimistic
Builds on: Dreamer RSSM LBSGD Opax CPO
Closest papers by meaning
- Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms (NeurIPS 2023 · similarity 0.90)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.89)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.89)
- Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.88)
- Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.