Research map / Large language models / Alignment and preferences
Rule Based Rewards for Language Model Safety
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rule-Based Rewards (RBR), which combine LLM-graded proposition probabilities from few-shot prompts with a helpful-only reward model during PPO, to improve safety while reducing over-refusals with minimal human data.
Problem
Human preference data for LLM safety is costly, slow to update, and under-specified instructions lead annotators to induce over-refusal or judgmental refusal styles.
Approach
Decompose desired safety behavior into binary propositions judged by a few-shot grader LLM, define rules over proposition truth values, fit a linear rule-based reward on synthetic ranked completions with a hinge loss, and add this reward to the helpful RM directly during PPO rather than distilling rules into RM data.
Key results
- RBR-PPO achieves human-evaluated F1 of 97.1 vs 91.7 for Human-PPO and 95.8 for Helpful-PPO (Large models).
- On WildChat unsafe prompts, RBR-PPO raises Not-Unsafe to 96.03% from 69.34% for Helpful-PPO, while Human-PPO's over-refusal on XSTest-style comply prompts drops to 84.7% human Not-Overrefuse vs 98.1% for RBR-PPO.
- Human-matched RBR-PPO with 518 human completions performs slightly worse than RBR-PPO, suggesting RBRs need less human data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with RBRs instead of human safety preference data | PPO safety training · Internal safety, human evaluation | F1-Score: 91.7 $\pm$ 1.3% -> 97.1 $\pm$ 0.7% | Helped |
| train with RBRs instead of helpful-only preferences | PPO safety training · Internal safety, human evaluation | F1-Score: 95.8 $\pm$ 0.8% -> 97.1 $\pm$ 0.7% | Helped |
| add RBR rewards to helpful-only PPO | Large policy model · MMLU | score: $75.9\pm0.8\%$ -> $74.4\pm0.9\%$ | Hurt |
| add RBR reward to outdated-safety-data training | PPO safety training · Internal safety, medium policy model | F1-Score: 87.7 $\pm$ 1.2% -> 94.2 $\pm$ 0.8% | Helped |
| add RBR reward to Human-RM training | PPO safety training · Internal safety, medium policy model | Not-Overrefuse: +16% | Helped |
| add synthetic safety SFT data but omit RBR reward (ablation) | RBR-PPO · Internal safety, medium policy model | F1-Score: 94.1 $\pm$ 0.8% -> 92.3 $\pm$ 0.9% | Hurt |
| omit the helpful RM score on safety prompts (ablation) | RBR-PPO · Internal safety, medium policy model | F1-Score: 94.1 $\pm$ 0.8% -> 92.7 $\pm$ 0.9% | Hurt |
| omit synthetic safety SFT data (ablation) | RBR-PPO · Internal safety, medium policy model | F1-Score: 94.1 $\pm$ 0.8% -> 92.2 $\pm$ 0.9% | Hurt |
| use only 518 human-labeled safety completions (ablation) | Human safety data baseline · Internal safety, medium policy model | F1-Score: 94.1 $\pm$ 0.8% -> 81.5 $\pm$ 1.5% | Hurt |
| increase the amount of safety prompts seen during PPO (ablation) | RBR-PPO · Internal safety | Not-Unsafe | Helped |
| increase the Hard-Refusal/Comply prompt ratio (ablation) | RBR-PPO · Internal safety | Not-Unsafe | Mixed |
| increase Soft-Refusal prompts from about one quarter to about matching Comply prompts (ablation) | RBR-PPO · Soft-Refusal prompts | Soft-Refusal style | Helped |
Surprising findings
- Human-PPO over-refused on every Comply prompt in an initial run due to imbalance of refusal examples in RM data, despite high safety scores.
- Helpful-PPO improved safety over Helpful-SFT even though helpful-only data contains no safety examples, hypothesized to stem from politeness correlation.
Limitations
- Best suited to behaviors that can be decomposed into explicit, easy-to-judge propositions; harder for subjective tasks like essay quality.
- Relies on LLM grader accuracy and may amplify LLM biases; correlated errors from repeated RBR usage in evaluation.
- Experiments use internal models and proprietary prompts, limiting reproducibility of absolute numbers.
Experimental setup
Datasets: Safety-relevant RL prompts (internal, 6.7K) Synthetic RBR comparison data (internal) Gold set (518 human-labelled completions) WildChat XSTest
Benchmarks: Internal safety RBR eval XSTest WildChat unsafe subset MMLU HellaSwag GPQA Lambada
Metrics: Not-Unsafe Not-Overrefuse F1 score between Not-Unsafe and Not-Overrefuse Hard-Refusal-Style ModAPI safety score Llama Guard 2 safety score error rate
Baselines: Helpful-PPO Human-PPO Human-match RBR-PPO Old Data-PPO HumanRM+RBR-PPO
Builds on: PPO Constitutional AI Sparrow rule-conditioned reward model
Closest papers by meaning
- RuleAdapter: Dynamic Rules for training Safety Reward Models in RLHF (ICML 2025 · similarity 0.91)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.90)
- Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models (ICLR 2025 · similarity 0.90)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.89)
- Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback (NeurIPS 2025 · similarity 0.89)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.