Research map / Large language models / Alignment and preferences
ENCORE: Entropy-guided Reward Composition for Multi-head Safety Reward Models
Xiaomin Li, Xupeng Chen, Jingxuan Fan, Eric Hanchen Jiang, Mingye Gao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add entropy-penalized rule weighting instead of uniform weights | Multi-head reward model, Llama3.1-8B · RewardBench safety | accuracy (%): 85.5 -> 88.5 | Improved |
| add entropy-penalized weighting instead of random weights | Multi-head reward model, Llama3.1-8B · RewardBench safety | accuracy (%): 84.2 -> 88.5 | Improved |
| use learned MoE gating instead of entropy weighting | Multi-head reward model, Llama3.1-8B · RewardBench safety | accuracy (%): 88.5 -> 86.0 | Hurt |
| use all 10 rules with multi-head instead of a single rule | Multi-head reward model, Llama3.1-8B · RewardBench safety | accuracy (%): 76.4 -> 85.5 | Improved |
| use multi-head reward model instead of single-head Bradley-Terry | Llama3.1-8B · RewardBench safety | accuracy (%): 66.61 -> 85.5 | Improved |
| use entropy weighting instead of Bradley-Terry | Llama3.1-8B · RewardBench XSTest Should Respond | accuracy (%): 13.6 -> 72.4 | Improved |
| use entropy weighting instead of MoE gating | Llama3.1-8B · RewardBench XSTest Should Respond | accuracy (%): 73.6 -> 72.4 | Hurt |
| use entropy weighting instead of uniform weights on alternate backbone | FsFairX-Llama3-8B · RewardBench safety | accuracy (%): 77.7 -> 83.1 | Improved |
| use entropy weighting instead of Bradley-Terry + FsfairX | FsFairX-Llama3-8B · RewardBench safety | accuracy (%): 79.3 -> 83.1 | Improved |
| select top 5 lowest-entropy rules (hard selection) instead of soft entropy weighting over all rules (ablation) | ENCORE · RewardBench safety | accuracy | Hurt |
| use an 8B ENCORE reward model instead of LLM-as-a-judge with GPT-3.5 and Llama models | Llama3.1-8B · RewardBench safety | accuracy (%): 64.0 -> 88.5 | Improved |
Closest papers by meaning
- Rule Based Rewards for Language Model Safety (NeurIPS 2024 · similarity 0.90)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.89)
- RuleAdapter: Dynamic Rules for training Safety Reward Models in RLHF (ICML 2025 · similarity 0.89)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.89)
- WARM: On the Benefits of Weight Averaged Reward Models (ICML 2024 · similarity 0.89)
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
