Research map / Label-efficient and robust learning / Recognition and long-tail
Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning reward models from ordinal (multi-level) human feedback under a marginal unbiasedness ("wisdom of the crowd") assumption, proving reduced Rademacher complexity versus binary feedback and validating with Llama-3.2-1B and Gemma-2-2B experiments on Skywork preference data.
Problem
Reward models for LLM alignment are typically trained from binary pairwise preferences, discarding tied pairs and fine-grained strength information that human annotators provide.
Approach
Introduces a marginal unbiasedness assumption (E[Z | x,y1,y2] = oracle preference probability) that lets ordinal labels be defined as random two-point distributions over a feedback scale, trains reward models with a generalized cross-entropy objective on these soft ordinal labels, and proves that finer ordinal feedback yields smaller Rademacher complexity via a hierarchical expectation coupling ar
Key results
- Across Llama and Gemma reward models, OOD accuracy on RewardBench rises with feedback granularity (Llama: 0.7667 binary vs 0.8193 oracle; Gemma: 0.8237 binary vs 0.8697 oracle).
- 5-level feedback closely tracks oracle performance (Llama OOD 0.8100 vs oracle 0.8193).
- With 32,768 samples, mixing 25-75% tied samples under the proposed objective improves over 0% tied (Gemma ID accuracy 0.9355 at 0% tied vs 0.9526 at 50% tied); 100% tied collapses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace binary feedback with 5-level feedback | Llama-3.2-1B-Instruct reward modeling · ID | accuracy: 0.9329 -> 0.9372 | Helped |
| Replace binary feedback with 5-level feedback | Llama-3.2-1B-Instruct reward modeling · OOD | accuracy: 0.7667 -> 0.8100 | Helped |
| Replace binary feedback with 5-level feedback | Gemma-2-2B-IT reward modeling · ID | accuracy: 0.9368 -> 0.9371 | No clear effect |
| Replace binary feedback with 5-level feedback | Gemma-2-2B-IT reward modeling · OOD | accuracy: 0.8237 -> 0.8584 | Helped |
| Add 25% tied samples to the training data (ablation) | Llama-3.2-1B-Instruct reward modeling · ID | accuracy: 0.9224 -> 0.9341 | Helped |
| Add 25% tied samples to the training data (ablation) | Gemma-2-2B-IT reward modeling · ID | accuracy: 0.9355 -> 0.9467 | Helped |
| Add 25% tied samples to the training data (ablation) | Gemma-2-2B-IT reward modeling · OOD | accuracy: 0.8319 -> 0.8487 | Helped |
| Use only tied samples (100% tied) (ablation) | Reward modeling · ID | accuracy: 0.9224 -> 0.3428; 0.9355 -> 0.4814 | Hurt |
Surprising findings
- Hinge loss variant performs weaker than cross-entropy in the experiments, attributed to margin tuning and sparse support vectors.
- Training on 100% tied data collapses reward learning to near-chance accuracy.
Limitations
- Oracle labels are synthetic, derived from a reward model rather than human annotators, with a tuned temperature T=20/3.
- Results rely on only two small base models and a single dataset plus RewardBench OOD evaluation.
- Hinge loss results are limited to one base model with coarse margin tuning.
Experimental setup
Datasets: Skywork-Reward-Preference-80K-v0.2 RewardBench
Benchmarks: RewardBench
Metrics: accuracy oracle cross-entropy loss
Baselines: Binary feedback Bradley-Terry with ties (BTT) Rao-Kupper model
Builds on: Bradley-Terry model DPO knowledge distillation soft labeling
Closest papers by meaning
- Reward Learning from Multiple Feedback Types (ICLR 2025 · similarity 0.90)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.89)
- Preference Learning with Response Time: Robust Losses and Guarantees (NeurIPS 2025 · similarity 0.87)
- Scaling Reward Modeling without Human Supervision (NeurIPS 2026 · similarity 0.87)
- ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning (NeurIPS 2025 · similarity 0.87)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.