Research map / Reinforcement learning / RL from feedback
Information-Theoretic Reward Decomposition for Generalizable RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes the implicit reward gap into prompt-free and prompt-related parts via an information-theoretic objective (binary search, no extra models) and prioritizes training pairs with small prompt-free gaps to improve reward model generalization.
Problem
Standard Bradley-Terry reward models can rely on response-only features (e.g., length) and ignore prompts, so they generalize poorly to unseen prompt-response pairs.
Approach
Extract a prompt-free reward component from the trained reward model via an MI-based criterion (solved by binary search with importance sampling over prompts, no extra model), then prioritize training samples with small prompt-free reward gaps using a binary-clustering plus EMA threshold.
Key results
- On Reward-Bench with LLaMA-3-8B, the method reaches 0.82 average accuracy vs 0.72 for vanilla BT and 0.77 for RRM.
- With Mistral-7B, average Reward-Bench accuracy is 0.75 vs 0.66 vanilla and 0.70 RRM.
- DPO with the method reaches 40.6 LC win rate on AlpacaEval-2 vs 30.5 vanilla and 39.5 RRM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prioritize training samples using prompt-free reward gaps | Reward model trained on RLHFlow preference data with LLaMA-3-8B-Instruct · Reward-Bench | Average accuracy: 0.77 -> 0.82 | Helped |
| Prioritize training samples using prompt-free reward gaps | Reward model trained on RLHFlow preference data with Mistral-7B-Instruct · Reward-Bench | Average accuracy: 0.70 -> 0.75 | Helped |
| Use the proposed reward model instead of RRM | DPO policy alignment with LLaMA-3-8B-Instruct · AlpacaEval-2, DPO | LCWR: 39.5 -> 40.6 | Helped |
| Use the proposed reward model instead of RRM | DPO policy alignment with LLaMA-3-8B-Instruct · AlpacaEval-2, DPO | WR: 40.9 -> 42.3 | Helped |
| Use the proposed reward model instead of RRM | Best-of-N alignment with LLaMA-3-8B-Instruct, N=4 · AlpacaEval-2, BoN (N=4) | LCWR: 35.3 -> 38.5 | Helped |
| Use the proposed reward model instead of RRM | Best-of-N alignment with LLaMA-3-8B-Instruct, N=4 · AlpacaEval-2, BoN (N=4) | WR: 38.4 -> 45.0 | Helped |
| Use the proposed reward model instead of RRM | Best-of-N alignment with LLaMA-3-8B-Instruct, N=32 · AlpacaEval-2, BoN (N=32) | LCWR: 41.1 -> 44.7 | Helped |
| Use the proposed reward model instead of RRM | Best-of-N alignment with LLaMA-3-8B-Instruct, N=32 · AlpacaEval-2, BoN (N=32) | WR: 44.9 -> 48.3 | Helped |
| Use the proposed reward model instead of RRM | DPO policy alignment with LLaMA-3-8B-Instruct · MT-Bench, DPO | T1: 8.35 -> 8.44 | Helped |
| Use the proposed reward model instead of RRM | DPO policy alignment with LLaMA-3-8B-Instruct · MT-Bench, DPO | T2: 7.46 -> 8.03 | Helped |
| Prioritize samples with smaller prompt-free reward gaps | Reward model trained on the length-biased dataset · Reward-Bench, length-biased data | Average: 50.6 -> 55.8 | Helped |
| Prioritize samples with smaller prompt-free reward gaps | Reward model trained on adversarial-prompt data · Reward-Bench, adversarial-prompt data | Average: 46.4 -> 51.6 | Helped |
Surprising findings
- Simple expectation-based definitions of prompt-free reward are shown to fail, motivating the MI-based solution.
- On the adversarial-prompt dataset, vanilla training with prompt-free dominance degrades Reward-Bench reasoning scores (36.5 vs 49.0 original).
Limitations
- Decomposition relies on a specific additive form that may not hold for all reward models, so the rewards are not exactly accurate.
- Importance-sampling approximation of P(y1,y2|x) for general datasets uses a heuristic fixed probability p.
Experimental setup
Datasets: SHP RLHFlow pair preference dataset UltraFeedback Reward-Bench
Benchmarks: Reward-Bench AlpacaEval-2 MT-Bench
Metrics: accuracy length-controlled win rate win rate MT-Bench score
Baselines: vanilla BT reward training RRM DPO Best-of-N
Builds on: Bradley-Terry model RRM DPO
Closest papers by meaning
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.90)
- A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.90)
- The Trickle-down Impact of Reward Inconsistency on RLHF (ICLR 2024 · similarity 0.89)
- InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling (NeurIPS 2024 · similarity 0.89)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.89)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.