Research map / Large language models / Alignment and preferences
RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RM-Bench, a reward model benchmark that uses same-LM chosen/rejected pairs with subtle errors and style-controlled variants (concise, plain detailed, markdown) to test substance sensitivity and style-bias robustness across chat, code, math and safety.
Problem
Existing reward model benchmarks pair responses from models of different strength, which fails to test sensitivity to subtle content changes and robustness to style bias, and correlates poorly with policy model performance.
Approach
Generate chosen and rejected responses from the same LM (gpt-4o) with subtle injected errors, and create three style variants per response (concise, detailed plain, detailed markdown) to form a 3x3 style-substance matrix with easy/normal/hard accuracy metrics.
Key results
- Skywork-Reward-Llama-3.1-8B achieves 70.1% average accuracy but only 46.6% hard accuracy on RM-Bench.
- Nemotron-4-340B-Reward reaches 69.5% average accuracy on RM-Bench.
- RM-Bench reward-model performance correlates with downstream policy performance at Pearson r=0.55 (p=0.07), versus r=0.21 for RewardBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate reward models on subtle content differences and style-controlled response pairs | RM-Bench · Reward model performance versus policy model downstream tasks | Pearson correlation: 0.55 ($p=0.07$) vs. 0.21 ($p=0.51$) | Helped |
| Use DPO with a reference model instead of a sequence classifier | Tulu-v2.5 reward modeling · HH-RLHF | Average Accuracy: 60.1 -> 62.1 | Helped |
| Use DPO with a reference model instead of a sequence classifier | Tulu-v2.5 reward modeling · StackExchange | Average Accuracy: 56.9 -> 59.9 | Helped |
| Use DPO with a reference model instead of a sequence classifier | Tulu-v2.5 reward modeling · Nectar | Average Accuracy: 54.1 -> 58.8 | Helped |
| Use DPO with a reference model instead of a sequence classifier | Tulu-v2.5 reward modeling · Chatbot Arena 2023 | Average Accuracy: 52.2 -> 57.5 | Helped |
| Remove the reference model from DPO reward scoring (ablation) | Tulu-v2.5 DPO models · HH-RLHF | Average Accuracy: 62.1 -> 54.4 | Hurt |
| Remove the reference model from DPO reward scoring (ablation) | Tulu-v2.5 DPO models · StackExchange | Average Accuracy: 59.9 -> 53.6 | Hurt |
| Remove the reference model from DPO reward scoring (ablation) | Tulu-v2.5 DPO models · Nectar | Average Accuracy: 58.8 -> 44.6 | Hurt |
| Remove the reference model from DPO reward scoring (ablation) | Tulu-v2.5 DPO models · Chatbot Arena 2023 | Average Accuracy: 57.5 -> 47.8 | Hurt |
| Increase reward model Hard Accuracy on RM-Bench Chat | Reward models used to train policy models · Arena-Hard-Auto style-controlled evaluation | policy model style-control score | Helped |
| Increase reward model performance on RM-Bench | Reward models used for Best-of-N selection · HumanEval and MATH | relative accuracy improvement | Helped |
Surprising findings
- Most reward models score at or below random (50%) on hard accuracy, i.e. they behave more like style-preference models.
- DPO models outperform their sequence-classifier counterparts trained on the same preference data, largely due to the reference model.
Limitations
- Only covers length and markdown style biases; other biases (e.g., code-only responses, phrases) are not covered.
- Correlation with policy models is shown only in a controlled setting with one base model (tulu-2-13b) and PPO, and with a small number of points (four reward models).
- Chosen and rejected responses are all generated by gpt-4o, which may bias evaluation toward gpt-4o style.
Experimental setup
Datasets: AlpacaEval XSTest HumanEvalPack MATH donotanswer AI2 Refusal GSM8K HumanEval+ MBPP+ ToxiGen Big Bench Hard Arena-Hard-Auto
Benchmarks: RM-Bench RewardBench
Metrics: accuracy easy accuracy normal accuracy hard accuracy Pearson correlation
Baselines: RewardBench
Builds on: RewardBench Many-Shot Jailbreaking Chatbot Arena style control Tulu 2.5
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.93)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.92)
- Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? (ICLR 2025 · similarity 0.92)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.92)
- WARM: On the Benefits of Weight Averaged Reward Models (ICML 2024 · similarity 0.92)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.