Research map / Reinforcement learning / RL from feedback
ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ResponseRank, which ranks pairwise comparisons by inferred preference strength using locally stratified response-time or annotator-agreement signals under a Plackett-Luce loss, plus the PDC metric for cardinal utility learning.
Problem
Binary preference labels in RLHF convey only preference direction, not preference strength, which is needed for decision-making under uncertainty and better reward model generalization; strength proxies like response times are noisy and confounded.
Approach
Stratify comparisons (e.g., per annotator), sort comparisons within each stratum by a noisy strength proxy, and train a utility model with a Plackett-Luce ranking loss over signed utility differences against a virtual zero anchor; reduces to Bradley-Terry for single-comparison strata.
Key results
- On MultiPref with inter-annotator agreement (RR-Agree), ResponseRank improves RewardBench 2 by +3.0 pp over BT and matches final BT RewardBench performance with about 30% fewer preference pairs.
- RR-RT (response-time stratified) performs poorly on MultiPref, indicating response time was not a useful strength signal in that dataset.
- On synthetic data, ResponseRank achieves significantly higher median PDC than BT and improves choice accuracy across deterministic, DDM and stochastic settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use inter-annotator agreement as the strength signal | ResponseRank on MultiPref · MultiPref test | accuracy | Helped |
| Use inter-annotator agreement as the strength signal | ResponseRank on MultiPref · RewardBench 2 | performance: +3.0{} pp improvement over BT | Helped |
| Use inter-annotator agreement as the strength signal | ResponseRank on MultiPref · RewardBench 2 | preference pairs to match final BT performance: approximately 30\% fewer preference pairs | Helped |
| Use response time stratified by annotator | ResponseRank on MultiPref · MultiPref | performance | Hurt |
| Use stated strength labels (slight/clear) instead of random strength rankings | ResponseRank on MultiPref · RewardBench 2 | performance: +0.6{} pp over BT; $p= 0.414$ | Helped |
| Use random strength rankings of size 2 (ablation) | ResponseRank on MultiPref · MultiPref test | accuracy | Helped |
| Use random strength rankings of size 2 (ablation) | ResponseRank on MultiPref · RewardBench 2 | performance | Hurt |
| Train with BT loss for two epochs instead of three (ablation) | BT on MultiPref · MultiPref test | performance | Helped |
| Train with BT loss for three epochs instead of two (ablation) | BT on MultiPref · RewardBench 2 | performance | Helped |
| Permute response times (ablation) | ResponseRank on synthetic preference datasets · Synthetic datasets | Pearson Distance Correlation (PDC) | Hurt |
| Permute response times (ablation) | ResponseRank on synthetic preference datasets · Synthetic datasets | choice accuracy | Hurt |
| Pool all comparisons into one global stratum instead of stratifying (ablation) | ResponseRank on synthetic preference datasets · Synthetic datasets | performance | Hurt |
Surprising findings
- Response times were not a useful strength signal on MultiPref, with RR-RT performing poorly.
- RR-Random, trained on random strength rankings, exceeds three-epoch BT on MultiPref test accuracy, though it does not generalize to RewardBench 2.
Limitations
- Response times were not informative on MultiPref, likely due to its complex annotation protocol.
- RL control experiments are small scale and differences often fall within confidence intervals.
- Tied preferences are discarded by the method.
Experimental setup
Datasets: MultiPref RewardBench 2 MuJoCo HalfCheetah MuJoCo Swimmer MuJoCo Walker2d Highway merge-v0 synthetic preference data
Benchmarks: MultiPref RewardBench 2 synthetic preference benchmarks
Metrics: Pearson Distance Correlation choice accuracy RewardBench 2 score policy return
Baselines: Bradley-Terry RtRegression ResponseRank-Perm ResponseRank-Pool BT@2 RR-Random
Builds on: Bradley-Terry Plackett-Luce BT-Regression
Closest papers by meaning
- Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries (ICML 2025 · similarity 0.91)
- Preference Learning with Response Time: Robust Losses and Guarantees (NeurIPS 2025 · similarity 0.91)
- Enhancing Preference-based Linear Bandits via Human Response Time (NeurIPS 2024 · similarity 0.90)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.90)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.