Research map / Large language models / Alignment and preferences
The Trickle-down Impact of Reward Inconsistency on RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contrast Instructions, a benchmark of lexically similar instruction pairs for measuring reward model consistency, shows standard LLaMa-7B reward models are near random on it, introduces ConvexDA (training augmentation) and RewardFusion (inference retrieval-based score fusion) to improve consistency, and shows an RLHF model trained with a more consistent RM gives more useful responses.
Problem
Reward models trained with the standard pairwise ranking objective may fail to recognize how semantic changes in the instruction should change which response is appropriate, and this reward inconsistency may degrade downstream RLHF chatbots.
Approach
Measure RM consistency with automatically built pairs of lexically similar instructions with different ground-truth responses (Contrast Instructions), then improve consistency without extra training cost via convex-hull-selected synonym data augmentation during training (ConvexDA) and similarity-weighted reward fusion with retrieved training examples at inference (RewardFusion).
Key results
- Standard LLaMa-7B reward models score 53.6 on response consistency and 49.4 on instruction consistency, near random (50), versus about 82 for human annotators.
- Combining ConvexDA and RewardFusion raises average consistency to 57.3 (response) and 54.1 (instruction), while the RLHF model with a Contrast-finetuned RM gets MT-Bench one-turn 3.89 vs 3.44 for the standard RM.
- Human evaluation shows usefulness rising from 50.9% to 57.8% with the more consistent RM, while relevance and factuality are essentially unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ConvexDA during training and RewardFusion during inference | LLaMa-7B reward model · Contrast Instructions (four-benchmark average) | Response consistency: 53.6 -> 57.3 | Helped |
| Apply ConvexDA during training and RewardFusion during inference | LLaMa-7B reward model · Contrast Instructions (four-benchmark average) | Instruction consistency: 49.4 -> 54.1 | Helped |
| Remove ConvexDA (ablation) | ConvexDA + RewardFusion reward model · Contrast Instructions (four-benchmark average) | Response consistency: 57.3 -> 55.6 | Hurt |
| Remove ConvexDA (ablation) | ConvexDA + RewardFusion reward model · Contrast Instructions (four-benchmark average) | Instruction consistency: 54.1 -> 53.2 | Hurt |
| Remove RewardFusion (ablation) | ConvexDA + RewardFusion reward model · Contrast Instructions (four-benchmark average) | Response consistency: 57.3 -> 56.5 | Hurt |
| Remove RewardFusion (ablation) | ConvexDA + RewardFusion reward model · Contrast Instructions (four-benchmark average) | Instruction consistency: 54.1 -> 53.7 | Hurt |
| Use a Contrast Ft. RM rather than a Standard RM during RLHF training | LLaMa-7B RLHF model · MT-Bench one-turn | score: 3.44 -> 3.89 | Helped |
| Use a Contrast Ft. RM rather than a Standard RM during RLHF training | LLaMa-7B RLHF model · MT-Bench multi-turn | score: 2.39 -> 3.18 | Helped |
| Use a more consistent RM rather than a Standard RM during RLHF training | LLaMa-7B RLHF model · StackExchange test questions | Usefulness: 50.9% -> 57.8% | Helped |
| Use a more consistent RM rather than a Standard RM during RLHF training | LLaMa-7B RLHF model · StackExchange test questions | Relevance: 92.4% -> 92.2% | No clear effect |
| Use a more consistent RM rather than a Standard RM during RLHF training | LLaMa-7B RLHF model · StackExchange test questions | Factuality: 49.1% -> 50.0% | No clear effect |
| Apply ConvexDA and RewardFusion | LLaMa-7B reward model · Original test sets (four-benchmark average) | RMEval: 75.6 -> 75.9 | No clear effect |
Surprising findings
- Standard RM evaluation (RMEval) barely changes even when consistency improves substantially, suggesting the original ranking metric does not capture reward consistency.
- Multi-task RM training did not transfer across the four tasks and did not improve consistency.
Limitations
- Gap to human consistency remains large even after the proposed techniques.
- Experiments use a single base model (LLaMa-7B) with LoRA and int8 training due to compute limits.
- Contrast Instructions does not cover all aspects of RM consistency or robustness.
Experimental setup
Datasets: StackExchange WMT Twitter RealSumm
Benchmarks: Contrast Instructions (StackExchange, WMT, Twitter, RealSumm) MT-Bench RMEval
Metrics: response consistency instruction consistency accuracy MT-Bench score human pairwise preference acceptability relevance/usefulness/factuality
Baselines: Standard RM Human Multi-task RM Vanilla data augmentation
Builds on: PPO StackLLaMa LoRA SimCSE
Closest papers by meaning
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.91)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.91)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.90)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.90)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
- A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.