Research map / Large language models / Alignment and preferences
RuleAdapter: Dynamic Rules for training Safety Reward Models in RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RuleAdapter, a Llama3.2-3B selector that dynamically picks the 5 most discrepant and relevant safety rules per response pair to label preferences, and trains an 8B reward model (RAMO) that tops the RewardBench safety leaderboard.
Problem
Fine-grained rule-based annotation for safety reward models requires choosing which rules to apply; applying all rules is inefficient and fixed or random subsets can be irrelevant or uninformative.
Approach
Select, per prompt and response pair, the r rules with the largest score discrepancy between the two responses plus a prompt-relevance term, justified as maximizing mutual information between rule-based labels and hidden ground-truth preferences, and learn this selector with a multi-label classifier (Rule Adapter).
Key results
- RAMO (8B) achieves a RewardBench safety score of 95.1, the highest on the leaderboard as of May 25, 2025, above SteerLM-70B (92.8) and Nemotron-340B (91.5).
- Ablation: the Rule Adapter selection reaches 95.1 vs 93.3 for dynamic random 5 rules, 92.9 for fixed 5 rules, and 93.9 for all 100 rules.
- On SafetyBench, Llama3.2-3B aligned with RAMO via PPO improves average safety from 76.7 to 77.4, and Llama3.2-1B from 54.2 to 56.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the reward model with dynamically selected critical rules using Rule Adapter rather than randomly sampling 5 rules (ablation) | RAMO · RewardBench-Safety | Safety score: 93.3 -> 95.1 | Helped |
| use dynamic Rule Adapter selection rather than a fixed random set of 5 rules (ablation) | RAMO · RewardBench-Safety | Safety score: 92.9 -> 95.1 | Helped |
| select 5 critical rules rather than apply all 100 rules (ablation) | RAMO · RewardBench-Safety | Safety score: 93.9 -> 95.1 | Helped |
| use the trained Rule Adapter rather than GPT-4 to dynamically select 5 rules (ablation) | RAMO · RewardBench-Safety | Safety score: 92.8 -> 95.1 | Helped |
| use rule-based preference labeling rather than direct GPT-4 preference labels (ablation) | RAMO · RewardBench-Safety | Safety score: 93.4 -> 95.1 | Helped |
| align Llama3.2-1B with PPO using RAMO | Llama3.2-1B · SafetyBench | Avg: 54.2 -> 56.8 | Helped |
| align Llama3.2-3B with PPO using RAMO | Llama3.2-3B · SafetyBench | Avg: 76.7 -> 77.4 | Helped |
Surprising findings
- Applying all 100 rules (93.9) outperforms dynamic GPT-4 rule selection (92.8) and GPT-4 direct preference labeling (93.4) in the ablation.
- Training on 1K RA-relabeled data beats human-labeled HH-RLHF data of matching size in most settings.
Limitations
- Fixed number of rules (r=5) rather than adaptive count.
- Rule pool and evaluation confined to the safety domain.
- Relies on synthetic LLM-judge labels; evaluation focuses on RewardBench-Safety with a small 1K training set.
Experimental setup
Datasets: ShareGPT HH-RLHF RewardBench-Safety SafetyBench
Benchmarks: RewardBench-Safety SafetyBench
Metrics: weighted safety score accuracy SafetyBench average accuracy
Baselines: SteerLM-70B Nemotron-340B Skywork-Llama3.1-8B URM QRM GPT-4 Claude 3.5 Gemini 1.5
Builds on: Constitutional AI Bradley-Terry reward modeling PPO SteerLM
Closest papers by meaning
- Rule Based Rewards for Language Model Safety (NeurIPS 2024 · similarity 0.91)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.90)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.90)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.90)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.