Research map / Large language models / Language, safety and interpretability
Generative Judge for Evaluating Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Auto-J, a 13B generative judge trained on GPT-4-annotated pairwise and single-response critiques across 58 real-world scenarios, supporting both pairwise comparison and natural-language critique with scores.
Problem
LLM evaluation needs to be general across real-world scenarios without references, flexible across protocols (pairwise and single-response), and interpretable with explanations, while open-source judges so far support only one protocol and give limited explanations.
Approach
Train a single open 13B judge on scenario-aware GPT-4 judgments (pairwise verdicts and critiques with ratings) built from real user queries across 58 scenarios, omitting criteria from the input so the model internalizes them, and using order-swap augmentation to reduce positional bias.
Key results
- Auto-J reaches 55.0% overall agreement with humans on Eval-P pairwise comparison, above ChatGPT (42.7) and Claude-2 (42.6), and below GPT-4 (62.3).
- On the AlpacaEval leaderboard, Auto-J's system-level ranking correlates with GPT-4's at Spearman 0.97 and Pearson 0.96.
- In Best-of-32 selection on LLaMA-2-Chat-7B, Auto-J's rating correlates with GPT-4 ratings at Pearson 0.57 versus 0.39 for a standard scalar reward model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Auto-J instead of open-source pairwise evaluators | Pairwise response evaluation · Eval-P | agreement rate | Helped |
| Use Auto-J instead of GPT-4 | Pairwise response evaluation · Eval-P | agreement rate | Hurt |
| Use Auto-J for single-response critique generation | Critique evaluation · Eval-C | critique preference | Helped |
| Use Auto-J ratings for best-of-N selection instead of baseline ratings | Single-response selection · Eval-R | GPT-4 rating: 8.20 -> 8.21; 8.14 -> 8.33; 8.14 -> 8.34; 7.51 -> 7.49; 7.69 -> 7.74; 7.66 -> 7.97 | Mixed |
| Use Auto-J ratings rather than baseline ratings | Response-level rating correlation · Eval-R | correlation with GPT-4 ratings: 0.41 -> 0.57 Pearson; 0.42 -> 0.55 Spearman | Helped |
| Add natural-language explanations to pairwise decisions (ablation) | Auto-J · Eval-P | agreement rate: 55.0 vs 54.8 | No clear effect |
| Use generative judgments instead of scalar reward outputs (ablation) | Auto-J · Eval-P | agreement rate: 54.5 vs 54.8 | Helped |
| Use Auto-J rather than a standard reward model for response selection (ablation) | Best-of-32 selection · LLaMA-2-Chat-7B | GPT-4 rating: 8.42 vs 8.34 | Hurt |
| Use Auto-J rather than a standard reward model for response selection (ablation) | Best-of-32 selection · Vicuna-7B-v1.5 | GPT-4 rating: 7.94 vs 7.97 | Helped |
| Use generative ratings instead of a standard scalar reward model (ablation) | Response-level rating correlation · Eval-R | Pearson correlation: 0.39 -> 0.57 | Helped |
| Use generative ratings instead of a standard scalar reward model (ablation) | Response-level rating correlation · Eval-R | Spearman correlation: 0.40 -> 0.55 | Helped |
Surprising findings
- Training with scenario criteria in the input made the model paraphrase the criteria, so criteria were omitted from inputs and learned implicitly via context distillation.
- Directly adding scenario criteria as the system message degraded GPT-4's single-response critiques, motivating a divide-and-conquer critique-merging step.
Limitations
- Training labels are GPT-4 judgments, so the judge inherits GPT-4 biases and is below GPT-4 in pairwise agreement.
- Eval-P human annotation and scenario classifier accuracy (72.55) are moderate, with a relatively small training set per scenario.
- Single-response critique tie rates diverge between GPT-4 and humans, suggesting evaluation signals are imperfect.
Experimental setup
Datasets: Chatbot Arena Conversations MTBench OpenAI Summary OpenAI WebGPT Stanford SHP Synthetic GPT-J PKU-SafeRLHF ShareGPT
Benchmarks: Eval-P Eval-C Eval-R AlpacaEval leaderboard
Metrics: agreement rate with human win rate average GPT-4 rating on best-of-N Pearson correlation Spearman correlation accuracy F1
Baselines: ChatGPT Claude-2 GPT-4 LLaMA-2-Chat-13B LLaMA-2-Chat-70B Vicuna-13B-v1.5 WizardLM-13B-v1.2 PandaLM SteamSHP SelFee Open-Assistant reward model
Builds on: LLaMA-2-Chat Context Distillation Ghost Attention MT-Bench
Closest papers by meaning
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.91)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.90)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.89)
- Exploring Intrinsic Alignments Within Text Corpus (AAAI 2025 · similarity 0.89)
- Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.