Research map / Reinforcement learning / RL from feedback
Safe Reinforcement Learning with Preference-based Constraint Inference
Chenglin Li, Grant Ruan, Hua Geng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dead zone (delta=1) to the safety loss (ablation) | PbCRL cost model · Safety Gymnasium Goal | W2 distance to true cost / ground-truth cost: W2 38.9 -> 16.8; cost 28.05 -> 18.50 | Improved |
| Increase dead zone from 1 to 2 (ablation) | PbCRL · Safety Gymnasium Goal | W2 distance / return: W2 16.8 -> 29.1; return 23.41 -> 22.51 | Hurt |
| Small dead zone delta=0.1 instead of 0 (ablation) | PbCRL · Safety Gymnasium Goal | W2 distance: 38.9 -> 33.5 | No clear effect |
| Lower the training safety threshold of PPO-BT (d=10) (ablation) | PPO-BT · Safety Gymnasium Goal | return / cost: cost 26.97 -> 14.02; return 24.26 -> 18.27 | Mixed |
| Add SNR loss with weight 1e-3 (ablation) | PbCRL · Safety Gymnasium Goal | mid-training cost: 24.8 -> 18.4 | Improved |
| Raise SNR loss weight to 1e-2 (ablation) | PbCRL · Safety Gymnasium Goal | end return: 23.4 -> 6.5 | Hurt |
| Add online finetuning with adaptive dead zone after offline pre-training (ablation) | PbCRL-Offline · Safety Gymnasium Goal | ground-truth cost: 22.70 -> 18.50 | Improved |
| Add online finetuning with adaptive dead zone after offline pre-training (ablation) | PbCRL-Offline · Safety Gymnasium Push | ground-truth cost: 37.48 -> 34.86 | Improved |
| Use PbCRL instead of standard BT cost model | PPO-BT · Safety Gymnasium Button | ground-truth cost: 50.78 -> 37.51 | Improved |
| Use PbCRL instead of segment-level binary cost (RLSF) | RLSF · Safety Gymnasium Button | return: -2.01 -> 8.73 | Improved |
| Use PbCRL instead of Safe RLHF | Safe RLHF · Safe RLHF dataset, Llama-3.2-1B | win rate (harmless): 76.5% -> 82.1% | Improved |
| Use PbCRL instead of PPO-BT | PPO-BT · Autonomous driving Block | cost: 0.12 ± 0.02 -> 0.07 ± 0.01 | Improved |
Closest papers by meaning
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.91)
- Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.90)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.90)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
