Research map / Large language models / Instruction and fine-tuning
A Critical Evaluation of AI Feedback for Aligning Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RLAIF's gains over SFT largely come from using a weaker teacher (GPT-3.5) for SFT data than the critic (GPT-4/Claude); plain SFT on the stronger critic's completions matches or beats the full RLAIF (SFT+DPO) pipeline.
Problem
Whether the two-step RLAIF pipeline (SFT then RL/DPO with AI feedback) is truly needed over simple SFT, given that popular open-source RLAIF models use a weaker teacher for SFT than the critic used for AI feedback.
Approach
Controlled comparison that matches the SFT teacher to the AI-feedback critic (GPT-4 or Claude) and shows that SFT on strong-teacher completions matches or outperforms RLAIF/DPO; adds a bandit analysis explaining when SFT beats RLAIF (weak student exploration limits DPO).
Key results
- SFT on GPT-4 completions consistently matches or outperforms RLAIF (GPT-3.5 SFT + GPT-4 feedback) across base models on AlpacaEval.
- RLAIF improves over GPT-3.5 SFT by over 10% absolute win rate for most base models with GPT-4 or Claude feedback, except Llama 7B with GPT-4 feedback.
- When the critic is also the SFT teacher (10% critic completions for SFT), the additional RLAIF gains are little to none over SFT alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mistral 7B rather than Llama 7B responses to generate preference data | RLAIF fine-tuning of Llama 7B · AlpacaEval | win rate: 45.0 (1.76) -> 47.6 (1.76) | No clear effect |
| Use Llama 7B rather than Mistral 7B responses to generate preference data | RLAIF fine-tuning of Mistral 7B · AlpacaEval | win rate: 71.1 (1.60) -> 68.5 (1.64) | No clear effect |
| Use GPT-4 completions instead of GPT-3.5 ShareGPT completions for SFT | SFT · AlpacaEval | win rate | Helped |
| Use Claude completions instead of GPT-3.5 ShareGPT completions for SFT | SFT · AlpacaEval | win rate | Helped |
| Use GPT-4 completions for both the initial SFT stage and preference data, rather than GPT-3.5 completions for SFT | RLAIF · AlpacaEval | win rate | No clear effect |
| Use Claude completions for both the initial SFT stage and preference data, rather than GPT-3.5 completions for SFT | RLAIF · AlpacaEval | win rate | No clear effect |
| Use GPT-4 or Claude AI feedback after SFT on GPT-3.5 ShareGPT completions | RLAIF · AlpacaEval | win rate | Helped |
| Use GPT-4 feedback for Llama 7B after SFT on GPT-3.5 ShareGPT completions | RLAIF · AlpacaEval | win rate | No clear effect |
| Train on preference data generated from another base model's responses | RLAIF fine-tuning · AlpacaEval | win rate | No clear effect |
Surprising findings
- Swapping preference datasets between Llama 7B and Mistral 7B leaves final performance close to original, suggesting base model matters more than preference data source.
- Using completions from the teacher as one side of preference pairs was found to give better models than sampling both from the SFT model.
Limitations
- Evaluation relies on AlpacaEval with GPT-4 or Claude as both critic and evaluator, which may favor the critic's preferences.
- Findings are shown for RLAIF with DPO and may not generalize to RLHF with human feedback or other RL algorithms.
- Bandit analysis is a synthetic simplified setting with 100 actions.
Experimental setup
Datasets: ShareGPT AlpacaEval
Benchmarks: AlpacaEval
Metrics: win rate
Baselines: SFT on GPT-3.5 ShareGPT completions RLAIF with GPT-4 critic RLAIF with Claude critic SFT on GPT-4 completions SFT on Claude completions
Builds on: DPO RLAIF AlpacaEval ShareGPT
Closest papers by meaning
- Interpreting Learned Feedback Patterns in Large Language Models (NeurIPS 2024 · similarity 0.91)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.90)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.90)
- Checklists Are Better Than Reward Models For Aligning Language Models (NeurIPS 2025 · similarity 0.89)
- Understanding the Effects of RLHF on LLM Generalisation and Diversity (ICLR 2024 · similarity 0.88)
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.