Research map / Large language models / Alignment and preferences
Ask a Strong LLM Judge when Your Reward Model is Uncertain
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes routing between a fast SNGP-based pairwise preference model and a costly DeepSeek-R1 LLM judge based on the PM's epistemic uncertainty, improving RM accuracy and RLOO-based online RLHF at equal judge-call budgets.
Problem
Classical reward models generalize poorly to OOD inputs and are vulnerable to reward hacking, while strong LLM judges are too slow and costly to call on every comparison during online RLHF.
Approach
Train a pairwise preference model with SNGP so its uncertainty can be computed in one pass; route only high-uncertainty response pairs to a strong LLM judge and use the resulting preference signal to build RLOO advantages.
Key results
- On RewardBench, uncertainty routing with 42.5% judge calls reaches 91.6% average accuracy vs 89.1% for random routing at the same count and 87.3% with no routing (DeepSeek-R1 on all: 92.3%).
- On RM-Bench, uncertainty routing at 44.1% calls reaches 81.9% average accuracy vs 77.1% for random routing; no routing gives 68.0%.
- In RLOO alignment, uncertainty routing (threshold 1.35, 6.6% calls) raises Arena-Hard WR to 28.9 vs 26.5 for random routing at similar calls (base 24.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Route uncertain pairs to DeepSeek-R1 rather than randomly selecting the same number of pairs | SNGP-PM on RewardBench · RewardBench | average accuracy: 89.1 -> 91.6 | Helped |
| Route uncertain pairs to DeepSeek-R1 rather than randomly selecting the same number of pairs | SNGP-PM on RM-Bench · RM-Bench | average accuracy: 77.1 -> 81.9 | Helped |
| Route uncertain pairs to DeepSeek-R1 rather than randomly selecting the same number of pairs | RLOO policy · Arena-Hard-v0.1 | win rate: 28.5 -> 29.8 | Helped |
| Route uncertain pairs to DeepSeek-R1 rather than randomly selecting the same number of pairs | RLOO policy · AlpacaEval 2.0 | length-controlled win rate: 25.98 -> 26.45 | Helped |
| Route uncertain pairs to DeepSeek-R1 rather than randomly selecting the same number of pairs | RLOO policy · MT-Bench | average score: 7.71 -> 7.71 | No clear effect |
| Add SNGP uncertainty estimation with a GP head instead of a standard linear head (ablation) | Preference model · HelpSteer2-Preference validation | accuracy: 0.801 -> 0.793 | No clear effect |
| Add SNGP uncertainty estimation with a GP head instead of a standard linear head (ablation) | Preference model · RewardBench | average accuracy: 0.877 -> 0.873 | No clear effect |
| Add SNGP uncertainty estimation with a GP head instead of a standard linear head (ablation) | Preference model · RM-Bench | average accuracy: 0.687 -> 0.680 | No clear effect |
| Add DeepSeek-R1 judge calls to the PM during RLOO | RLOO policy · Arena-Hard-v0.1 | win rate: 28.1 -> 29.8 | Helped |
| Add DeepSeek-R1 judge calls to the PM during RLOO | RLOO policy · AlpacaEval 2.0 | length-controlled win rate: 25.40 -> 26.45 | Helped |
Surprising findings
- Uncertainty routing requires more wall-clock inference time than random routing at equal call counts, indicating routed pairs are harder for the judge.
Limitations
- Judge is DeepSeek-R1 used via rate-limited API (200 RPM) and is not fine-tuned for judging; costly to host.
- Downstream alignment experiments are limited in scale (first 33% of UltraFeedback, K=4, one epoch) and the RL algorithm is only RLOO.
- Cost is measured in number of judge calls; SNGP overhead and iso-FLOP cost are not analyzed.
Experimental setup
Datasets: HelpSteer2-Preference UltraFeedback RewardBench RM-Bench Arena-Hard-v0.1 AlpacaEval 2.0 MT-Bench
Benchmarks: RewardBench RM-Bench HelpSteer2-Preference validation Arena-Hard-v0.1 AlpacaEval 2.0 MT-Bench
Metrics: accuracy win rate length-controlled win rate MT-Bench score number of judge calls inference time
Baselines: Random routing Standard pairwise PM (no routing) DeepSeek-R1 judge (full calls) Base Llama-3.1-8B-Instruct
Builds on: SNGP RLOO GRPO PPO OpenRLHF
Closest papers by meaning
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.90)
- On the Robustness of Reward Models for Language Model Alignment (ICML 2025 · similarity 0.90)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.90)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.89)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.89)
- Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.