Research map / Large language models / Alignment and preferences
Reward Modeling from Natural Language Human Feedback
Zongqi Wang, Rui Wang, Yuchuan Wu, Yiyao Yu, Pinyi Zhang, Shaoning Sun, Yujiu Yang, Yongbin Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add human-critique process reward to outcome reward (ablation) | RM-NLHF-Qwen-7B · 7-benchmark average | average accuracy: 0.6158 -> 0.6481 | Improved |
| add outcome regularization (no process reward when outcome is wrong) (ablation) | RM-NLHF-Qwen-7B · 7-benchmark average | average accuracy: 0.6282 -> 0.6481 | Improved |
| train MetaRM only on HelpSteer3 instead of also using outcome data D_o (ablation) | RM-NLHF-Qwen-7B · 7-benchmark average | average accuracy: 0.6332 -> 0.6481 | Improved |
| use online MetaRM instead of offline MetaRM (ablation) | MetaRM on Qwen-7B · MetaRM ablation overall | overall accuracy: 0.5493 -> 0.5552 | Improved |
| use online MetaRM instead of offline MetaRM (ablation) | MetaRM on Llama-8B · MetaRM ablation overall | overall accuracy: 0.6273 -> 0.6356 | Improved |
| add MetaRM process reward to outcome-only reward (No-MetaRM -> Online-MetaRM) (ablation) | GRM on Qwen-7B · MetaRM ablation overall | overall accuracy: 0.5347 -> 0.5552 | Improved |
| use only MetaRM reward without outcome signal instead of combining both (ablation) | MetaRM on Llama-8B · MetaRM ablation overall | overall accuracy: 0.6356 -> 0.6143 | Hurt |
| use only MetaRM reward without outcome signal instead of combining both (ablation) | MetaRM on Qwen-7B · MetaRM ablation overall | overall accuracy: 0.5552 -> 0.5579 | Mixed |
| update MetaRM aggressively (lr 1e-5, 2 epochs per round) (ablation) | MetaRM on Qwen-7B · MetaRM ablation overall | overall accuracy: 0.5552 -> 0.5407 | Hurt |
| discretize MetaRM output to binary instead of continuous regression (ablation) | MetaRM on Qwen-7B · MetaRM ablation overall | overall accuracy: 0.5552 -> 0.5469 | Hurt |
| use classification head instead of regression for MetaRM (ablation) | MetaRM on Qwen-7B · MetaRM ablation overall | overall accuracy: 0.5552 -> 0.5400 | Hurt |
| use similarity with core human-critique arguments instead of LLM-as-a-Meta-Judge for process reward (ablation) | process reward proxy (gpt-5-mini) · 49 annotated HelpSteer3 samples | accuracy: 0.4898 -> 0.8571 | Improved |
Closest papers by meaning
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.91)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.91)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.90)
- Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.90)
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.90)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
