Research map / Large language models / Alignment and preferences
Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes seq2seq reward modeling, replacing binary MLE reward training with sequence MLE over (rejected→chosen) language feedback, and extracts token-level positive/negative rewards for PPO, improving RLHF on safety, summarization and OOD prompts.
Problem
Scalar sequence-to-scalar reward models give coarse feedback, causing RLHF to exhibit unexpected generalization such as refusal-to-response paradigms and long-response bias.
Approach
Train the reward model as a seq2seq generator that maps rejected responses to chosen ones (correction mapping) plus identity mapping on chosen responses, then extract token-level positive and negative rewards by checking token agreement with the RM's generated output, without extra annotations or models.
Key results
- Seq2seq RM with PPO-T achieves an average win rate of 76.9% across 3 NLP tasks and 2B/7B models (PPO-T vs PPO in Table comparisons).
- On PKU-SafeRLHF with Llama2-7B, PPO-T reaches utility score -4.03 versus -12.23 for PPO, with a safety win rate of 0.81.
- On Dialogsum, PPO inflates response length by 27.31 tokens over Init-SFT (7B), while PPO-T changes length by only 1.66.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace seq2scalar RM with seq2seq RM using token-level rewards | PPO on Dialogsum · Dialogsum | response length (Δ vs Init-SFT): 162.17 (Δ 14.84) -> 149.16 (Δ 1.83) | Helped |
| Replace PPO with token-level seq2seq-RM PPO | Gemma-2B on Dialogsum · Dialogsum | GPT-4 win rate vs Init-SFT: 0.61 ± 0.24 -> 0.83 ± 0.14 | Helped |
| Replace PPO with token-level seq2seq-RM PPO | Llama2-7B on PKU-SafeRLHF · PKU-SafeRLHF | utility score: -12.23 ± 1.25 -> -4.03 ± 2.74 | Helped |
| Replace PPO with token-level seq2seq-RM PPO | Gemma-2B on PKU-SafeRLHF · PKU-SafeRLHF | safety win rate vs Init-SFT: 0.59 ± 0.24 -> 0.88 ± 0.11 | Helped |
| Use token-level seq2seq-RM PPO instead of PPO | Gemma-2B on OOD Xsum prompts · OOD Xsum prompts (RM trained on TL;DR) | win rate vs SFT: 56.7% ± 22.8% -> 75.2% ± 18.7% | Helped |
| Use only negative token scores, removing positive scores (ablation) | seq2seq-RM PPO on PKU-SafeRLHF · PKU-SafeRLHF | empty-output frequency: 68.5% (Gemma-2B); 86% (Llama2-7B) | Hurt |
| Add negative token scores to positive-only PPO-T (ablation) | seq2seq-RM PPO on PKU-SafeRLHF · PKU-SafeRLHF | alignment performance | Helped |
| Add positive token scores to negative-only PPO (ablation) | seq2seq-RM PPO on PKU-SafeRLHF · PKU-SafeRLHF | empty-output frequency: 68.5% (Gemma-2B); 86% (Llama2-7B) | Helped |
| Replace token-level rewards with sentence-level summed rewards (ablation) | PPO-T on alignment tasks · Dialogsum | GPT-4 win rate vs Init-SFT: 0.80 ± 0.16 (PPO-S, 2B) vs 0.83 ± 0.14 (PPO-T, 2B); 0.78 ± 0.17 (PPO-S, 7B) vs 0.81 ± 0.15 (PPO-T, 7B) | Mixed |
| Replace token-level seq2seq-RM PPO with direct seq2seq-RM correction of Init-SFT responses (ablation) | RLHF on PKU-SafeRLHF, Dialogsum, and TL;DR · PKU-SafeRLHF, Dialogsum, and TL;DR | win rate vs Init-SFT: 76.7% ± 17.9%; 75.2% ± 18.6%; 73.2% ± 12.2% (Gemma-2B); 82.2% ± 14.6%; 78.6% ± 16.8%; 79.3% ± 15.1% (Llama2-7B) | Hurt |
Surprising findings
- Using only negative scores causes LLMs to converge to generating no characters (68.5% for Gemma-2B, 86% for Llama2-7B).
- Applying the seq2seq RM directly as a corrector on Init-SFT responses does not surpass RLHF with PPO-T.
Limitations
- Evaluation relies heavily on GPT-4 judging and phenomenon analysis, with win rates showing high variance.
- Dialogsum preferences are synthetic GPT-4 annotations rather than human labels.
- Only two model sizes (Gemma-2B, Llama2-7B) and three tasks are tested.
Experimental setup
Datasets: PKU-SafeRLHF Dialogsum TL;DR Xsum
Benchmarks: PKU-SafeRLHF Dialogsum TL;DR Xsum MT Bench
Metrics: win rate utility score safety win rate response length difference
Baselines: PPO DPO RFT PPO-S
Builds on: PPO DPO RFT Bradley-Terry reward modeling
Closest papers by meaning
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.94)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.93)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.92)
- Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.92)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.92)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.