Research map / Large language models / Alignment and preferences
Training language models to follow instructions with human feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune GPT-3 with demonstrations and human preference feedback using PPO | GPT-3 · API prompt distribution | human preference: 85 ± 3% preferred to GPT-3; 71 ± 4% preferred to few-shot GPT-3 | Helped |
| Add updates on the pretraining mix during PPO (ablation) | PPO · API prompt distribution | labeler preference | No clear effect |
| Add updates on the pretraining mix during PPO (ablation) | PPO · SQuADv2, DROP, HellaSwag, and WMT 2015 French-to-English | public NLP task performance | Helped |
| Increase the KL coefficient instead of mixing pretraining updates (ablation) | PPO · DROP and SQuAD | public NLP task performance | Hurt |
| Fine-tune with human feedback | GPT-3 · Closed-domain API tasks | hallucination rate: 41% -> 21% | Helped |
| Fine-tune with human feedback | GPT-3 · TruthfulQA | truthfulness and informativeness: about twice as often | Helped |
| Add pretraining-mix updates to PPO at 1.3B parameters (ablation) | PPO · TruthfulQA | truthfulness | Hurt |
| Fine-tune with human feedback | GPT-3 · RealToxicityPrompts with respectful prompt | toxic output generation: about 25% fewer toxic outputs | Helped |
| Remove the respectful prompt (ablation) | InstructGPT · RealToxicityPrompts | toxicity | No clear effect |
| Prompt the model to produce toxic output (ablation) | InstructGPT · RealToxicityPrompts | toxicity | Hurt |
| Fine-tune with human feedback | GPT-3 · Winogender and CrowS-Pairs | bias | No clear effect |
| Add a respectful instruction (ablation) | InstructGPT · Winogender and CrowS-Pairs | bias | Hurt |
Closest papers by meaning
- Aligning Language Models with Demonstrated Feedback (ICLR 2025 · similarity 0.89)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (ICML 2024 · similarity 0.89)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.88)
- Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision (ICLR 2025 · similarity 0.88)
- Policy Improvement using Language Feedback Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.