Research map / Large language models / Alignment and preferences
On Evaluating LLM Alignment by Evaluating LLMs as Judges
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs' generation and evaluation rankings on alignment are strongly correlated under a strong GPT-4o oracle, and builds AlignEval, a benchmark that ranks LLMs by their evaluation accuracy against oracle labels without generating-and-judging outputs.
Problem
Evaluating LLM alignment with human preferences usually requires human annotators or LLM judges to score open-ended generated outputs, which is costly; it is unknown whether an LLM's evaluation capability tracks its generation capability at the ranking level.
Approach
Define generation-evaluation consistency (GE-consistency) as the rank correlation between LLMs' generation performance and their evaluation accuracy against a preference oracle, then use the reusable evaluation-task instances (filtered for oracle self-consistency) as a benchmark that ranks LLMs as judges without judging their outputs.
Key results
- GE-consistency reaches a Spearman correlation of 0.971 on Arena-Hard and 0.839 on AlpacaEval with GPT-4o as the preference oracle (after consistency filtering; 0.793 and 0.743 without filtering).
- Combining AlignEval with IFEval (AlignEval+) yields a Spearman correlation of 0.946 with style-controlled ChatBot Arena rankings across 23 LLMs, comparable to Arena-Hard (0.946) and GPT4o-Judge (0.958).
- Without IFEval, AlignEval-claude reaches 0.885 Spearman with style-controlled Arena rankings, versus 0.882 for Arena-Hard-SC and 0.761 for AlpacaEval, and it clearly outperforms MixEval (0.816).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine AlignEval-gpt with IFEval by averaging rankings | AlignEval-gpt · ChatBot Arena style-controlled rankings | Spearman correlation: 0.856 -> 0.946 | Helped |
| Combine AlignEval-claude with IFEval by averaging rankings | AlignEval-claude · ChatBot Arena style-controlled rankings | Spearman correlation: 0.885 -> 0.946 | Helped |
| Use consistency filtering for oracle judgments (ablation) | GE-consistency measurement with GPT-4o oracle · AlpacaEval | Spearman correlation: 0.743 -> 0.839 | Helped |
| Use consistency filtering for oracle judgments (ablation) | GE-consistency measurement with GPT-4o oracle · Arena-Hard | Spearman correlation: 0.793 -> 0.971 | Helped |
| Use Arena-Hard rather than AlpacaEval instructions | GE-consistency measurement with GPT-4o oracle · LLM generation-evaluation rankings | Spearman correlation: 0.839 -> 0.971 | Helped |
| Evaluate GE-consistency with GPT-4o as the preference oracle | LLM generation-evaluation consistency · Arena-Hard | Spearman correlation: 0.971 | Helped |
| Use AlignEval rather than MixEval to benchmark alignment | Automatic alignment benchmarks · ChatBot Arena style-controlled rankings | Spearman correlation: 0.816 -> 0.856; 0.816 -> 0.885 | Helped |
| Use HelpSteer3 annotations rather than AlignEval instances | LLM-as-judge benchmark · ChatBot Arena style-controlled rankings | Spearman correlation: 0.813 vs 0.856; 0.813 vs 0.885 | Hurt |
| Use AlignEval-claude rather than AlignEval-gpt | AlignEval · ChatBot Arena style-controlled rankings | Spearman correlation: 0.856 -> 0.885 | Helped |
| Use GPT4o-Judge rather than AlignEval-gpt | Automatic alignment benchmarks · ChatBot Arena style-controlled rankings | Spearman correlation: 0.856 -> 0.911 | Helped |
Surprising findings
- Consistency filtering matters greatly: removing it drops GE-consistency from 0.971 to 0.793 on Arena-Hard.
- Self-preference bias appears in AlignEval: AlignEval-gpt ranks gpt-4o-2024-05-13 second, while AlignEval-claude ranks claude-3.5-sonnet first.
Limitations
- AlignEval is a proxy evaluation and may be vulnerable to adversarial attacks such as fine-tuning a model to act as a judge.
- ChatBot Arena is used as the gold standard even though it is not a true gold standard given opaque data collection and biases.
- Using GPT-4o as oracle introduces self-preference bias; the GE-consistency depends strongly on the oracle's strength.
Experimental setup
Datasets: AlpacaEval Arena-Hard WildBench IFEval MixEval HelpSteer3 ChatBot Arena leaderboard
Benchmarks: AlignEval AlpacaEval Arena-Hard IFEval MixEval HelpSteer3 ChatBot Arena
Metrics: Spearman's rank correlation Cohen's Kappa win rate
Baselines: AlpacaEval Arena-Hard GPT4o-Judge IFEval MixEval HelpSteer3
Builds on: AlpacaEval Arena-Hard MT-Bench GV-consistency
Closest papers by meaning
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.92)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.92)
- Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.92)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.91)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.91)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.