Research map / Large language models / Alignment and preferences
RMB: Comprehensively benchmarking reward models in LLM alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RMB, a fine-grained reward model benchmark with pairwise and Best-of-N evaluations over real-world helpfulness and harmlessness scenarios, and checks its correlation with downstream BoN alignment.
Problem
Existing reward model benchmarks use coarse, limited-distribution pairwise accuracy that may not reflect RMs' effectiveness in downstream alignment (e.g., best-of-N guidance).
Approach
Build a fine-grained benchmark over 49 real-world scenarios (37 helpfulness, 12 harmlessness) from WildChat prompts, with GPT-4-turbo scenario-based scoring to form pairwise and Best-of-N test sets, and validate by Spearman correlation between benchmark rankings and BoN alignment results.
Key results
- GPT-4o achieves the top overall RMB score (0.738) among evaluated RMs, with Qwen2-72B-Instruct at 0.723 and Starling-RM-34B at 0.712.
- BoN accuracy is on average 17.5% lower than pairwise accuracy, with generative RMs dropping more than discriminative ones.
- Benchmark rankings correlate positively with BoN alignment rankings on external benchmarks, while RewardBench shows poor correlation; majority-vote confidence weighting did not improve correlation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate reward models with BoN instead of pairwise accuracy | Reward models on RMB · RMB helpfulness and harmlessness | accuracy: 17.5% | Hurt |
| add verbose evaluation criteria (ablation) | Generative RMs on helpfulness pairwise evaluation · RMB helpfulness | accuracy: -0.009; -0.049; -0.045; -0.081; -0.039; +0.003 | Hurt |
| add chain-of-thought reasoning (ablation) | Generative RMs on helpfulness pairwise evaluation · RMB helpfulness | accuracy: +0.018; -0.019; +0.104; +0.019; +0.012; -0.005 | Mixed |
| use majority-vote confidence-weighted evaluation instead of unweighted evaluation (ablation) | Reward models evaluated on RMB · Downstream alignment correlation | correlation | No clear effect |
Surprising findings
- Majority-voting confidence weighting did not improve correlation with downstream alignment performance.
- Helpfulness and harmlessness rankings are negatively correlated (Spearman -0.57) among top-10 RMs, indicating a trade-off.
Limitations
- Did not use full RL (e.g., PPO) to verify alignment correlation, using BoN instead.
- Generative RMs were not validated on BoN alignment due to excessive AI feedback cost and bias accumulation.
- Benchmark relies on LLM-generated responses, which may limit diversity.
Experimental setup
Datasets: WildChat MixEval Arena-Hard AdvBench RewardBench
Benchmarks: RMB RewardBench MixEval-hard Arena-Hard AdvBench
Metrics: pairwise accuracy best-of-n accuracy spearman rank correlation
Baselines: RewardBench
Builds on: RewardBench Best-of-N sampling LLM-as-a-judge
Closest papers by meaning
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.93)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.92)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.92)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.92)
- Interpreting Language Reward Models via Contrastive Explanations (ICLR 2025 · similarity 0.92)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.