Research map / Large language models / Alignment and preferences
Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures how much prompt instruction detail (none/perplexity, generic, criteria-name, full rubric) affects LLM-as-a-judge alignment with human judgements across 8 benchmarks and a 4-category criteria taxonomy, and compares against perplexity.
Problem
Whether LLM-as-a-judge scores reflect the provided evaluation instructions or the model's preference for its own fine-tuning-like data, and how much prompt detail improves alignment with human annotations.
Approach
Compares four prompting levels for LLM judges (perplexity without prompt, generic quality prompt, criterion-name prompt, full rubric) and proposes a prompt-free perplexity baseline, organizing 34 criteria from 8 benchmarks into a four-category taxonomy (Content, Relevance, Integrity, Engagement).
Key results
- Across the taxonomy, moving from generic prompt to full rubric improves average Pearson correlation by only about 4%, with GPT4-Turbo at 0.414 (generic) vs 0.469 (full rubric).
- For content-based criteria, perplexity reaches Pearson 0.51 versus 0.44 for prompting the LLM judge, and perplexity beats prompting by over 20% on average for content criteria.
- On logical correctness (Flask), GPT-4 reaches Pearson 0.68 versus 0.34 for Llama3 70b and 0.33 for Phi3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full rubric instead of a criteria-specific prompt (ablation) | Phi3-Medium · All datasets, all criteria | Pearson correlation: 0.367 -> 0.334 | Hurt |
| Use full rubric instead of a generic quality prompt (ablation) | Llama3 8b · All datasets, all criteria | Pearson correlation: 0.256 -> 0.352 | Helped |
| Use full rubric instead of a criteria-specific prompt (ablation) | GPT4-Turbo · All datasets, all criteria | Pearson correlation: 0.468 -> 0.469 | No clear effect |
| Use perplexity instead of generic quality prompting | LLM evaluators · Content-based criteria | Pearson correlation: 0.44 -> 0.51 | Helped |
| Use perplexity instead of generic prompting (ablation) | LLM evaluators · Flask | Pearson correlation: 0.365 -> 0.448 | Helped |
| Use perplexity instead of generic prompting (ablation) | LLM evaluators · Summeval | Pearson correlation: 0.345 -> 0.408 | Helped |
| Use full rubric instead of perplexity (ablation) | LLM evaluators · TheNextChapter | Pearson correlation: 0.275 -> 0.340 | Helped |
| Use full rubric instead of perplexity (ablation) | LLM evaluators · Roscoe | Pearson correlation: 0.159 -> 0.392 | Helped |
| Use full rubric instead of generic quality prompting (ablation) | LLM evaluators · Hanna | Pearson correlation: 0.232 -> 0.318 | Helped |
| Use full rubric instead of criteria-specific prompting (ablation) | LLM evaluators · TopicalChat | Pearson correlation: 0.426 -> 0.426 | No clear effect |
| Use full rubric instead of criteria-specific prompting (ablation) | LLM evaluators · OpinSummEval | Pearson correlation: 0.342 -> 0.328 | Hurt |
| Use full rubric instead of generic prompting (ablation) | LLM evaluators · Content-based naturalness | Pearson correlation: 0.454 -> 0.466 | Helped |
Surprising findings
- Perplexity often aligns better with human judgements than prompting LLM judges, especially on textual quality criteria.
- Detailed rubric prompts give little benefit for most models and can even hurt performance (e.g., Phi3 on the main results table).
Limitations
- Taxonomy category boundaries are acknowledged as not perfectly separated, with criteria overlapping across groups.
- Evaluation relies on Pearson correlation with existing human annotations from prior benchmarks rather than new human judgements.
- Only textual tasks are evaluated, and the reported dataset-level averages may be biased by text-quality criteria.
Experimental setup
Datasets: SummEval TopicalChat OpinSummEval InstruSumm Hanna TheNextChapter Roscoe Flask
Benchmarks: SummEval TopicalChat OpinSummEval InstruSumm Hanna TheNextChapter Roscoe Flask
Metrics: Pearson correlation
Baselines: GPT4-Turbo GPT3.5-Turbo Llama3 70b Llama3 8b Mistral Phi3-Medium Prometheus-2
Builds on: LLM-as-a-judge G-Eval Prometheus
Closest papers by meaning
- Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025 · similarity 0.92)
- Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations (NeurIPS 2025 · similarity 0.90)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.90)
- LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (AAAI 2025 · similarity 0.89)
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.89)
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.