Research map / Large language models / Alignment and preferences

Scaling Reward Modeling without Human Supervision

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains reward models without human labels by treating true next-text continuations as chosen and in-batch mismatched continuations as rejected, using 11M tokens of math web text, and evaluates on RewardBench v1/v2, best-of-N, and GRPO.

Problem

Reward models usually need costly and noisy human preference labels; the paper asks whether useful reward signals can be learned from raw, uncurated web text alone.

Approach

Converts raw web documents into online preference pairs by splitting each sequence into prefix and suffix, treating the true continuation as chosen and other in-batch continuations as implicit negatives, then training with a Bradley-Terry loss plus a score-centering regularizer.

Key results

• Trained RMs improve RewardBench v2 average by up to +7.7 points over initialization, with +16.1 on in-domain math subsets.
• Centering loss improves RewardBench v2 peak gain from +6.4 to +7.7 and GSM8K BoN delta MAP from +2.8 to +5.2 on Llama-3.1-8B-Instruct actor.
• Continuing curated RLHFMix1 training from the web-trained RM gives +10.3 average RewardBench v2 gain, with OOD-Safety +24.4 vs +22.6 for Llama-3.2-3B-Instruct.

Surprising findings

• Unsupervised web-text RMs reach OOD-Safety gains comparable to or above instruct initialization on RewardBench v2.
• Removing centering can cause BoN accuracy to fall below the random baseline at large N.

Limitations

• Experiments restricted to math-focused web text; generality to other domains untested.
• Absolute RewardBench v2 scores remain well below strong curated-data RMs such as Skywork.
• Results are reported with limited seeds and mostly single-run comparisons.

Experimental setup

Datasets: FineMath-4plus InfiwebMath-4plus RLHFMix1 (UltraFeedback, HH-RLHF, SHP, Summarize-from-Feedback) GSM8K MATH

Benchmarks: RewardBench v1 RewardBench v2 MATH500 GSM8K Toxigen IFEval

Metrics: accuracy average score delta MAP mean@1 test accuracy

Baselines: Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-3B random candidate selection randomly initialized seed RM

Builds on: Bradley-Terry GRPO RewardBench verl

Closest papers by meaning

• Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.91)
• GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.90)
• Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.90)
• Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models (NeurIPS 2026 · similarity 0.89)
• Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.89)
• Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.89)