Research map / Large language models / Reasoning and chain of thought
Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvalPlanner, a self-training method that teaches LLM judges to generate unconstrained evaluation plans and execute them step-by-step, optimized via SFT and two DPO iterations on ~22K synthetic preference pairs.
Problem
LLM-as-a-Judge models need chain-of-thought rationales for evaluation, but no human-annotated reasoning traces exist, and prior methods rely on hand-designed rubrics or criteria that do not transfer across domains.
Approach
Decouple the judge's chain-of-thought into an unconstrained, instruction-specific evaluation plan and a plan execution that yields the verdict, then optimize both jointly with iterative DPO over correct/incorrect (plan, execution, verdict) triples built from synthetic preference pairs.
Key results
- Achieves 93.9 overall on RewardBench with Llama-3.1-70B-Instruct seed, using 22K synthetic preference pairs, a new SOTA among generative reward models.
- Reaches 56.6 (Llama-3.1-70B seed) and 65.4 (Llama-3.3-70B seed) on FollowBenchEval, about 13 points above Skywork-Critic-Llama-3.1-70B.
- Second DPO iteration improves RewardBench accuracy from 92.3 to 93.9, while a single iteration on the same data gives only 92.3 to 92.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train EvalPlanner with two DPO iterations using updated-model data rather than one iteration (ablation) | EvalPlanner · RewardBench | accuracy: 92.3 -> 93.9 | Helped |
| use 22K preference pairs in one DPO iteration instead of 5K (ablation) | EvalPlanner · RewardBench | accuracy: 92.3 -> 92.5 | Helped |
| add DPO preference optimization of plans and executions after SFT with thoughts (ablation) | EvalPlanner · RewardBench | accuracy: 86.8 -> 90.5 | Helped |
| add intermediate thoughts to training (ablation) | Llama-3.1-70B-Instruct · RewardBench | accuracy: 84.0 -> 86.2 | Helped |
| use unconstrained plans instead of criteria-list plans (ablation) | EvalPlanner · RewardBench | accuracy: 83.9 -> 86.8 | Helped |
| use unconstrained plans instead of verification-question plans (ablation) | EvalPlanner · RewardBench | accuracy: 84.8 -> 86.8 | Helped |
| increase sampled plans and executions from 3 plans and 4 executions to 5 plans and 8 executions (ablation) | EvalPlanner · RewardBench | accuracy (Overall): 88.8 -> 90.5 | Helped |
| train with WildChat rather than MATH prompts (ablation) | EvalPlanner · RewardBench Chat-Hard | accuracy: 79.8 -> 82.9 | Helped |
| train with MATH rather than WildChat prompts (ablation) | EvalPlanner · RewardBench Reasoning | accuracy: 89.3 -> 93.4 | Helped |
| use EvalPlanner with Llama-3.3-70B-Instruct as seed instead of Skywork-Critic-Llama-3.1-70B | FollowBenchEval · FollowBenchEval | position-consistent accuracy (Overall): 52.2 -> 65.4 | Helped |
| use EvalPlanner with Llama-3.3-70B-Instruct seed instead of Skywork-Critic-Llama-3.1-70B | RM-Bench · RM-Bench | accuracy (Overall): 74.1 -> 82.1 | Helped |
| train the judge with two DPO iterations rather than one iteration on 22K pairs (ablation) | EvalPlanner · RewardBench | accuracy: 92.5 -> 93.9 | Helped |
Surprising findings
- Unconstrained, self-generated evaluation plans outperform plans constrained to hand-designed criteria or verification questions (86.8 vs 83.9 and 84.8 on RewardBench ablation).
Limitations
- Iterative DPO gains were shown for only two iterations; further iterations left for future work.
- Relies on a 70B seed model; 8B results are lower on RewardBench (83.0 vs 84.1 for 70B judge).
Experimental setup
Datasets: WildChat MATH RewardBench PPE RM-Bench JudgeBench FollowBench
Benchmarks: RewardBench PPE RM-Bench JudgeBench FollowBenchEval
Metrics: accuracy position-consistent accuracy
Baselines: Self-Taught Evaluator Skywork-Critic-Llama-3.1-70B SFR-Llama-3.1-70B-Judge LMUnit Critic-RM-Rank DeepSeek-GRM-27B Llama-3.1-70B-Instruct Llama-3.3-70B-Instruct Claude-3.5-Sonnet GPT-4o Gemini-1.5-pro-0514
Builds on: Direct Preference Optimization Self-Taught Evaluators Chain-of-Verification
Closest papers by meaning
- J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (ICLR 2026 · similarity 0.91)
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.90)
- Think-J: Learning to Think for Generative LLM-as-a-Judge (AAAI 2026 · similarity 0.90)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.89)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.88)
- Characterizing, Evaluating, and Optimizing Complex Reasoning (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.