Research map / Large language models / Alignment and preferences
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dense sentence-level reward instead of holistic sequence-level reward | RLHF on GPT-2 large · RealToxicityPrompts test | toxicity avg max: 0.130 -> 0.081 | Helped |
| use dense sentence-level reward instead of holistic reward | RLHF on GPT-2 large · RealToxicityPrompts test | perplexity: 11.75 -> 9.77 | Helped |
| use fine-grained multi-reward RLHF instead of preference RLHF | T5-large QA policy · QA-Feedback test | R2 factuality reward: 0.781 -> 0.816 | Helped |
| use fine-grained multi-reward RLHF instead of preference RLHF | T5-large QA policy · QA-Feedback test | R1 relevance reward: 0.482 -> 0.513 | Helped |
| use fine-grained multi-reward RLHF instead of preference RLHF | T5-large QA policy · QA-Feedback test | RougeLSum: 49.84 -> 49.93 | No clear effect |
| RLHF (either type) vs SFT for relevance errors | T5-large QA policy · QA-Feedback human eval | relevance error rate | No clear effect |
| remove relevance reward model R1 (ablation) | F.G. RLHF · QA-Feedback dev | RougeLSum: 38.52 -> 44.29 | Helped |
| add factuality reward model R2 (ablation) | F.G. RLHF · QA-Feedback dev | R2 factuality reward: 0.640 -> 0.825 | Helped |
| add completeness reward model R3 (ablation) | F.G. RLHF · QA-Feedback dev | RougeLSum: 43.61 -> 44.29 | Helped |
| increase relevance reward weight w1 (0.2 -> 0.4) (ablation) | F.G. RLHF · QA-Feedback test | R1 relevance reward: 0.425 -> 0.637 | Mixed |
| fine-grained completeness reward model vs preference reward model | reward model · QA-Feedback pairwise accuracy | accuracy: 57.2 -> 70.9 | Helped |
Closest papers by meaning
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.92)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.91)
- Interpreting Learned Feedback Patterns in Large Language Models (NeurIPS 2024 · similarity 0.91)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.91)
- RLVF: Learning from Verbal Feedback without Overgeneralization (ICML 2024 · similarity 0.91)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.