Research map / Large language models / Alignment and preferences
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori Hashimoto
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize a learned surrogate reward with PPO instead of stopping at SFT | LLaMA 7B · AlpacaFarm evaluation, human | win-rate against Davinci003: 44% to 55% | Improved |
| Use more supervised fine-tuning data, SFT 52k instead of SFT 10k | LLaMA 7B · AlpacaFarm evaluation | win-rate against Davinci003: 36.7\pm1.7 to 39.2\pm1.7 (simulated); 44.3\pm1.7 to 40.7\pm1.7 (human) | No clear effect |
| Use Binary FeedME on preferred responses rather than SFT alone | LLaMA 7B · AlpacaFarm evaluation | win-rate against Davinci003: 36.7\pm1.7 -> 36.6\pm1.7 (simulated); 44.3\pm1.7 -> 37.9\pm1.7 (human) | No clear effect |
| Use binary reward conditioning on pairwise preferences rather than SFT alone | LLaMA 7B · AlpacaFarm evaluation, simulated | win-rate against Davinci003: 36.7\pm1.7 -> 32.4\pm1.6 | Hurt |
| Use PPO rather than Expert Iteration to optimize the surrogate reward | LLaMA 7B · AlpacaFarm evaluation | win-rate against Davinci003: 41.9\pm1.7 -> 46.8\pm1.8 (simulated); 45.7\pm1.7 -> 55.1\pm1.7 (human) | Improved |
| Increase best-of-n sampling from the SFT model | LLaMA 7B · AlpacaFarm evaluation | win-rate against Davinci003: 36.7\pm1.7 -> 45.0\pm1.7 (simulated); 44.3\pm1.7 -> 50.7\pm1.8 (human) | Improved |
| Train with the top reward bin and KL regularization in Quark | LLaMA 7B · AlpacaFarm evaluation, simulated | win-rate against Davinci003: 36.7\pm1.7 -> 35.6\pm1.7 | Hurt |
| Add label-flip noise to simulated preference labels (ablation) | AlpacaFarm training annotator · Best-of-n reward optimization | simulated win-rate as sample count increases: Label-flipping ratios tested: 0\%, 17.5\%, 25\%, 35\% | Mixed |
| Use a pool of 13 annotators instead of a single deterministic GPT-4 prompt (ablation) | AlpacaFarm preference simulator · Reward-model over-optimization | win-rate as reward optimization increases | Improved |
| Use label noise and a pool of annotators to reproduce human variability (ablation) | AlpacaFarm preference simulator · Reward-model over-optimization | win-rate as reward optimization increases | Improved |
| Use a single low-variance GPT-4 annotator rather than the variable AlpacaFarm annotator pool for simulated PPO training (ablation) | PPO, LLaMA 7B · Human evaluation | win-rate: 43% -> 50% | Improved |
| Normalize PPO advantages across the full rollout batch rather than separately for each minibatch (ablation) | PPO implementation · PPO training stability | training stability | Improved |
Closest papers by meaning
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (ICML 2024 · similarity 0.89)
- OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.88)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.88)
- AlpaGasus: Training a Better Alpaca with Fewer Data (ICLR 2024 · similarity 0.88)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.87)
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
