Research map / Reinforcement learning / RL from feedback
Pairwise Calibrated Rewards for Pluralistic Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning a small ensemble of reward functions whose mixture pairwise-matches annotator preference fractions (pairwise calibration), via a forward stagewise additive modeling heuristic on soft labels, with theory on size, outliers and generalization.
Problem
Standard RLHF with a single Bradley-Terry reward collapses diverse human preferences into a majority signal, losing minority viewpoints and causing preference collapse in pluralistic settings.
Approach
Model preferences as a small weighted ensemble of reward functions that is pairwise calibrated: for each response pair, the fraction of ensemble rewards preferring one response matches the fraction of annotators doing so. Ensemble is learned without annotator IDs by fitting each new reward to the residual calibration error (FSAM) using soft vote-fraction labels.
Key results
- An ensemble of 2-4 weak reward models already achieves noticeably better held-out calibration MSE than the theoretically optimal single deterministic reward on many datasets, with gains tapering around six models (k=8 ensembles fit).
- Theory: an O(1/epsilon)-sized ensemble achieves epsilon pairwise calibration error, and an outlier-free sub-ensemble remains nearly calibrated.
- Pairwise Kendall-tau between reward models is low (highest diversity on Reddit TL;DR), indicating the reward models rank responses differently.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an FSAM ensemble of reward models instead of a single deterministic reward model | Meta-Llama-3-8B reward models · Held-out prompts across MultiPref, PersonalLLM, HelpSteer2, and Reddit TL;DR | mean-squared calibration error | Helped |
Surprising findings
- Training on binary majority labels discards useful soft-label information; the paper argues soft-label BCE matches the true BT log-likelihood.
- Computing an exact pairwise-calibrated ensemble is NP-hard even with a single context and known pairwise fractions, via reduction from minimum feedback arc set.
Limitations
- Pairwise calibration does not guarantee accurate representation of higher-order (multi-response) preferences.
- Calibration is defined at reward-ordering level, ignoring the KL-regularized policy probabilities, which is only a tractable proxy.
- Pairwise calibration may be inappropriate for highly contested topics where neutral or refusal responses are preferred.
Experimental setup
Datasets: MultiPref PersonalLLM HelpSteer2 Reddit TL;DR PRISM
Benchmarks: MultiPref PersonalLLM HelpSteer2 Reddit TL;DR
Metrics: mean-squared calibration error normalized Kendall-tau correlation between reward rankings
Baselines: optimal single deterministic reward model
Builds on: Bradley-Terry model RLHF forward stagewise additive modeling
Closest papers by meaning
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.92)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.90)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.89)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
- No Preference Left Behind: Group Distributional Preference Optimization (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.