Research map / Reinforcement learning / Visual and representation RL
Safety through feedback in Constrained RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLSF, an on-policy constrained RL method that infers a binary cost function from segment-level safety feedback using a state-level surrogate classification loss and novelty-based query sampling.
Problem
Constrained RL when the cost function is unknown and must be inferred from costly evaluator feedback (human or system), especially for complex, long-horizon environments such as self-driving.
Approach
Elicit binary safe/unsafe feedback on trajectory segments, convert the segment-level likelihood into a state-level surrogate binary classification loss with noisy labels, and select trajectories for feedback only when they contain enough novel (SimHash-unseen) states, which implicitly yields a decreasing query schedule.
Key results
- RLSF with inferred costs reaches within about 80% of the best PPOLag run (known costs) in 7 of 11 benchmark environments, and has lower cost violation than SIM and SDM across benchmarks.
- On Point Circle RLSF achieves return 36.42 with C.V rate 1.9% versus SDM's 11.43% and SIM's 35.21%.
- Novelty-based sampling uses about 1950 queries on average (vs. around 10^4 trajectories for baselines given every trajectory), and outperforms uniform/decreasing random and entropy sampling schedules.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use novelty-based trajectory sampling instead of random or entropy sampling with uniform or decreasing schedules | RLSF · Biased Pendulum and benchmark environments | performance | Helped |
| reduce the feedback buffer to 100k (ablation) | RLSF · benchmark environments | policy performance | Hurt |
| increase the feedback segment length to trajectory-level feedback rather than state-level feedback (ablation) | RLSF · Point Circle versus Point Goal | cost inference accuracy | Mixed |
| optimize the surrogate loss instead of the MLE loss (ablation) | RLSF · trajectory-level feedback | gradient stability | Helped |
| increase the safety threshold by a constant δ to compensate for overestimation bias (ablation) | RLSF · Car Circle | performance | Helped |
| use autoencoders for state feature extraction in SimHash rather than raw states (ablation) | RLSF novelty sampling · state hashing | performance | No clear effect |
| use transferred inferred cost rather than the true cost | PPOLag · Doggo Circle and Doggo Goal | return: 2.69 ± 0.24 -> 2.00 ± 0.27; 1.45 ± 0.06 -> 1.20 ± 0.26 | Mixed |
Surprising findings
- Transferring the inferred cost function from a Point agent to a Doggo agent gives PPOLag performance comparable to the true cost in Circle and Goal tasks.
- Segment-level feedback induces an overestimation bias in the inferred cost, leading to overly conservative policies; adding a constant bonus to c_max improves return in Car Circle.
Limitations
- Relies on state-level feedback in some environments (sparse-violation tasks), which is expensive to obtain.
- Feedback is simulated from the true cost function; real noisy human feedback is not tested.
- Segment-level feedback causes overestimation bias; the bias-correction heuristic delta lacks a safety guarantee and needs tuning.
Experimental setup
Datasets: Safety Gymnasium (Point/Car Circle, Goal, Push; Doggo) Mujoco-based Biased Pendulum, Blocked Swimmer, HalfCheetah, Hopper, Walker2d Driver simulator (blocking, lane change, two-lane overtaking)
Benchmarks: Safety Gymnasium Mujoco safety tasks (Biased Pendulum, Blocked Swimmer, HalfCheetah, Hopper, Walker2d) Driver simulator
Metrics: return cost violation rate cost
Baselines: PPO-Lagrangian SIM with known costs (SIMKC) Self Imitation Safe RL (SIM) Safe Distribution Matching (SDM) PPO
Builds on: PPO-Lagrangian Constrained MDP Self-Imitation Safe RL (SIM) SimHash
Closest papers by meaning
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.88)
- Evaluating Model-Free Reinforcement Learning toward Safety-Critical Tasks (AAAI 2023 · similarity 0.87)
- Plan-R1: Safe and Feasible Trajectory Planning as Language Modeling (ICLR 2026 · similarity 0.87)
- RuleAdapter: Dynamic Rules for training Safety Reward Models in RLHF (ICML 2025 · similarity 0.86)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Building a Foundational Guardrail for General Agentic Systems via Synthetic Data (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.