Research map / Large language models / Alignment and preferences
ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds UltraFeedback, a ~1M-feedback AI (GPT-4) preference and critique dataset over 250k conversations, and uses it to train UltraRM, UltraCM, and PPO-tuned UltraLM-13B-PPO.
Problem
Open-source chat LLM alignment is bottlenecked by the small size and narrow topical coverage of human preference datasets, which limits feedback learning research.
Approach
Scale and diversity of AI feedback: broad instructions from six sources, completions from 17 models with varied system principles, and GPT-4 annotation using four-aspect decomposition, score documentation, multi-completion reference scoring, and rationales, plus textual critiques.
Key results
- UltraRM (LLaMA2-13B) reaches 71.0% average reward-modeling accuracy across four human preference datasets, the best open-source reward model reported, behind only the much larger closed LLaMA2 helpfulness RM.
- UltraLM-13B-PPO achieves a 69.7% average win rate across AlpacaEval, Evol-Instruct and UltraChat, the highest among the compared open models, improving over UltraLM-13B by 16.8 points.
- Best-of-16 sampling with UltraRM raises UltraLM-13B's AlpacaEval win rate from 76.53% (single sample) to 91.54%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train UltraRM with fine-grained UltraFeedback scores instead of overall scores (ablation) | UltraRM · OpenAI WebGPT | accuracy: 62.0 -> 65.2 | Helped |
| train UltraRM with only UltraFeedback, without mixing open-source preference datasets (ablation) | UltraRM · OpenAI WebGPT | accuracy: 65.1 vs 65.2 | No clear effect |
| use PPO alignment with UltraFeedback feedback | UltraLM-13B · AlpacaEval, Evol-Instruct, and UltraChat | average win rate: +16.8 percent enhancement | Helped |
| select the highest-reward response from 2 samples instead of 1 | UltraLM-13B · AlpacaEval | win rate: 76.53% -> 84.64% | Helped |
| select the highest-reward response from 16 samples instead of 1 | UltraLM-13B · AlpacaEval | win rate: 76.53% -> 91.54% | Helped |
| apply PPO alignment | UltraLM-13B · Capability benchmarks average | exact match: 63.3 -> 64.3 | Helped |
| apply PPO alignment | UltraLM-13B · BoolQ | exact match: 85.0 -> 83.5 | Hurt |
Surprising findings
- Reward model trained only on UltraFeedback already outperforms open-source baselines by over 6 points, and mixing in other datasets adds further gains.
- GPT-4 agreement with the human majority vote (68.6%) is higher than with individual annotators (about 60%), suggesting GPT-4 reflects collective human preference.
Limitations
- Relies on GPT-4 as annotator and judge, which cannot model human preference precisely in all situations.
- Only single-turn dialogues are included; the dataset does not cover multi-turn conversation.
- Weaker on math and code tasks, and little safety-oriented data was included intentionally.
Experimental setup
Datasets: UltraFeedback Stanford SHP OpenAI Summarization Anthropic Helpful TruthfulQA FalseQA Evol-Instruct UltraChat ShareGPT FLAN
Benchmarks: AlpacaEval Evol-Instruct UltraChat OpenAI WebGPT OpenAI Summarization Anthropic HH-RLHF Stanford SHP BoolQ HellaSwag RACE MultiRC TriviaQA
Metrics: accuracy win rate win/tie/lose rate exact match critique score (1-7) agreement rate
Baselines: Moss Ziya OASST reward model SteamSHP LLaMA2-70B-Chat WizardLM-13B-v1.2 Vicuna-13B-v1.5 SelFee Shepherd
Builds on: PPO UltraLM Constitutional AI MT-Bench
Closest papers by meaning
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.88)
- RLVF: Learning from Verbal Feedback without Overgeneralization (ICML 2024 · similarity 0.88)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.88)
- Human Feedback is not Gold Standard (ICLR 2024 · similarity 0.88)
- CollabLLM: From Passive Responders to Active Collaborators (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.