Research map / Large language models / Alignment and preferences
Checklists Are Better Than Reward Models For Aligning Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reinforcement Learning from Checklist Feedback (RLCF), which extracts instruction-specific checklists, scores responses with an AI judge and verifier programs, and uses the weighted scores for DPO preference tuning of Qwen2.5-7B-Instruct.
Problem
Reinforcement learning for instruction following relies on fixed criteria or reward models that are arbitrary or hackable; automatic, flexible, and objective feedback for open-ended instructions is lacking.
Approach
Generate instruction-specific checklists of yes/no requirements (candidate-response-based), grade each response per item with an LLM judge (averaged over 25 samples) and code verifiers where applicable, then combine weighted scores into rewards for DPO on response pairs.
Key results
- RLCF is the only method that improves on every benchmark, with a 4-point boost in FollowBench hard satisfaction rate and a 6-point InFoBench gain over Qwen2.5-7B-Instruct per the abstract.
- On IFEval loose prompt-level accuracy, DPO (RLCF) reaches 77.3 vs 75.0 for Qwen2.5-7B-Instruct.
- Off-policy RLCF improves Llama 3.1 8B Instruct on InFoBench overall (83.1 to 84.2) and FollowBench HSR (68.0 to 72.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply candidate-based checklist feedback with DPO | Qwen2.5-7B-Instruct · InFoBench Overall | score: 78.1 -> 84.1 | Helped |
| Apply candidate-based checklist feedback with DPO | Qwen2.5-7B-Instruct · FollowBench Hard Satisfaction Rate, Avg | hard satisfaction rate: 71.4 -> 75.3 | Helped |
| Apply candidate-based checklist feedback with DPO | Qwen2.5-7B-Instruct · Arena-Hard Vanilla | win rate: 51.3 -> 54.6 | Helped |
| Apply candidate-based checklist feedback with DPO | Qwen2.5-7B-Instruct · AlpacaEval Length-Controlled | win rate: 36.2 -> 37.1 | Helped |
| Use Skywork reward feedback for DPO instead of checklist feedback | Qwen2.5-7B-Instruct · AlpacaEval Length-Controlled | win rate: 36.2 -> 41.5 | Helped |
| Use Skywork reward feedback for DPO instead of checklist feedback | Qwen2.5-7B-Instruct · IFEval (prompt), Avg | score: 77.3 -> 76.0 | Hurt |
| Generate checklists directly from instructions instead of using candidate responses (ablation) | RLCF on Qwen2.5-7B-Instruct · IFEval (prompt), Avg | score: 78.6 -> 76.9 | Hurt |
| Remove program verification (ablation) | RLCF on Qwen2.5-7B-Instruct · FollowBench Avg hard satisfaction rate | hard satisfaction rate: 75.3 -> 73.8 | Hurt |
| Remove prompt-based scoring (ablation) | RLCF on Qwen2.5-7B-Instruct · FollowBench Avg hard satisfaction rate | hard satisfaction rate: 75.3 -> 73.6 | Hurt |
| Remove program verification (ablation) | RLCF on Qwen2.5-7B Base · Arena-Hard Vanilla | win rate: 19.4 -> 23.1 | Helped |
| Apply RLCF | Qwen2.5-7B-Instruct · GSM8K | accuracy: 83.2 -> 82.2 | Hurt |
| Apply off-policy RLCF | Llama 3.1 8B Instruct · FollowBench Hard Satisfaction Rate | hard satisfaction rate: 68.0 -> 72.3 | Helped |
Surprising findings
- Off-the-shelf reward models (Skywork, ArmoRM) improve InFoBench and AlpacaEval but cause regressions on IFEval and FollowBench, while RLCF is the only consistent improver.
- Checklist-based rewards are worse than specialized reward models on RewardBench overall, yet yield better downstream RL performance.
Limitations
- Relies on a 72B teacher judge (strong-to-weak generalization) and is computationally expensive (~4 days on 8 H100s).
- Only preference-based (DPO) RL explored; policy-gradient methods not tested.
- Weaker on safety and slight regressions on GSM8K and TruthfulQA, attributed to WildChat prompt distribution.
Experimental setup
Datasets: WildChat WildChecklists InFoBench IFEval FollowBench AlpacaEval Arena-Hard RewardBench
Benchmarks: IFEval InFoBench FollowBench AlpacaEval Arena-Hard RewardBench XSTest GSM8K TruthfulQA
Metrics: loose/strict accuracy InFoBench requirement satisfaction ratio FollowBench soft/hard satisfaction rate constraint satisfaction level win rate length/style-controlled win rate accuracy
Baselines: SFT (distilled) Skywork-Reward-Gemma-2-27B DPO ArmoRM-Llama3-8B-v0.1 DPO UltraFeedback DPO AI Judge DPO
Builds on: DPO UltraFeedback AutoIF
Closest papers by meaning
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.90)
- Generalizing Verifiable Instruction Following (NeurIPS 2025 · similarity 0.89)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.89)
- BLEUBERI: BLEU is a surprisingly effective reward for instruction following (NeurIPS 2025 · similarity 0.89)
- A Critical Evaluation of AI Feedback for Aligning Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.