Research map / Large language models / Alignment and preferences
Long-form RewardBench: Evaluating Reward Models for Long-form Generation
Hui Huang, Yancheng He, Wei Liu, Muyun Yang, Jiaheng Liu, Kehai Chen, Bing Xu, Conghui Zhu, Hailong Cao, Tiejun Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate generative reward models in Selection mode instead of Scoring mode (ablation) | generative reward models · Long-form RewardBench | overall accuracy: gemini-2.5-flash 40.8 -> 73.6; gpt-4o 41.1 -> 60.9 | Improved |
| use sequence classifiers instead of generative models for overall long-form reward modeling | reward models · Long-form RewardBench | overall accuracy: top classifier 74.9 vs best generative 74.6 | Improved |
| use generative models (with chain-of-thought) instead of classifiers on reasoning subset | reward models · Long-form RewardBench Reasoning | pairwise accuracy: classifiers about 70% vs generative over 90% (gemini-2.5-flash 90.2) | Improved |
| use fine-tuned generative judges (RISE, Selene, Skywork-Critic) for best-of-n and pointwise scoring | fine-tuned generative judges · Long-form RewardBench BoN subsets | valid output rate: invalid up to 73.9% (Skywork-Critic Scoring) | Hurt |
| increase response length from 1K to 8K tokens in needle test (ablation) | generative RM (GPT-4o) · Long-form Needle-in-a-Haystack | error detection accuracy | Hurt |
| place the error in the middle of the response (ablation) | generative RM (GPT-4o) · Long-form Needle-in-a-Haystack | error detection accuracy | Hurt |
| increase response length (ablation) | sequence classifier (Skywork-Llama-3.1-8B-v0.2) · Long-form Needle-in-a-Haystack | error detection accuracy | No clear effect |
| place the error near the end of the response (ablation) | sequence classifier · Long-form Needle-in-a-Haystack | error detection accuracy | Improved |
| train on longer preference data (ablation) | classifier reward models · Long-form RewardBench, 30k synthetic pairs split by length | accuracy | No clear effect |
| train on longer preference data (ablation) | generative reward models · Long-form RewardBench, 30k synthetic pairs split by length | accuracy | Improved |
| use Qwen base instead of Llama base for generative reward model (ablation) | generative reward models · Long-form RewardBench | accuracy | Improved |
Closest papers by meaning
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.91)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.91)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.89)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.89)
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.89)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
