Research map / Large language models / Alignment and preferences
Chain of Hindsight aligns Language Models with Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Chain of Hindsight (CoH), which fine-tunes language models on sequences of model outputs paired with templated natural-language feedback (good/bad) so they learn from all feedback without RL.
Problem
Aligning language models with human preferences via SFT wastes negatively rated data, while RLHF relies on imperfect reward models and difficult RL optimization.
Approach
Convert all human preference rankings into templated hindsight feedback sequences (e.g., "Bad: {worse} Good: {better}") and fine-tune with the standard causal LM objective, with loss only on non-feedback tokens, so the model learns from positive and negative examples and generates the good behavior when prompted with "Good".
Key results
- In human pairwise evaluation on summarization, CoH achieves a 57.5% average win rate over the pretrained base model and 45.3% vs RLHF (30.8% for RLHF over CoH's tie/loss split in the reported table).
- Human pairwise evaluation on dialogue gives CoH a 36.9% average win rate vs RLHF, with CoH winning 39.4% vs SFT and 56.0% vs SFT-U.
- On the few-shot alignment tax benchmark, CoH-tuned GPT-J averages 40.95 zero-shot vs 40.54 for SFT and 40.60 for base GPT-J.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Chain of Hindsight instead of RLHF | Language models on TL;DR summarization · TL;DR summarization | human evaluation average win rate (%): 30.8 -> 57.5 | Helped |
| Use Chain of Hindsight instead of RLHF | Language models on HH dialogue · HH dialogue | human evaluation average win rate (%): 23.4 -> 49.5 | Helped |
| Add natural language feedback (ablation) | Chain of Hindsight · TL;DR summarization | human preference win rate (%): 10.6 -> 15.1 | Helped |
| Add natural language feedback (ablation) | Chain of Hindsight · TL;DR summarization | human preference win rate (%) against RLHF: 42.4 -> 45.3 | Helped |
| Add unlikelihood loss on negative examples | SFT on HH dialogue · HH dialogue | preferred dialogue classification accuracy | Hurt |
| Increase model size | Chain of Hindsight on summarization · Summarization benchmark | performance | Mixed |
| Condition training on multiple feedback-example pairs rather than only binary feedback (ablation) | Chain of Hindsight · TL;DR summarization | human evaluation preference | Helped |
Surprising findings
- Standard SFT on positively rated data decreased average few-shot performance after alignment, while CoH showed modest improvements over the base model.
- Adding unlikelihood loss (SFT-U) degraded dialogue preference performance relative to SFT, though it was still outperformed by CoH.
Limitations
- Constructing CoH sequences with multiple feedback instances yields long sequences that increase training cost.
- Evaluation relies on hired human labelers, which is costly.
- Uses templated feedback derived from ratings rather than open-ended human feedback.
Experimental setup
Datasets: WebGPT comparisons Anthropic HH-RLHF OpenAI summarize_from_feedback TL;DR Reddit ShareGPT The Pile
Benchmarks: TL;DR summarization (ROUGE and human evaluation) Anthropic HH dialogue preference classification few-shot NLP benchmarks (Language Model Evaluation Harness)
Metrics: ROUGE classification accuracy human pairwise win rate (%) few-shot accuracy
Baselines: SFT SFT-U C-SFT RLHF Koala GPT-J
Builds on: RLHF Decision Transformer unlikelihood training hindsight experience replay
Closest papers by meaning
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.90)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.89)
- Interpreting Learned Feedback Patterns in Large Language Models (NeurIPS 2024 · similarity 0.88)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.88)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.