Research map / Large language models / AI and society
Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that LLM-as-judge (LLJ) evaluation is premature, critically assessing four assumptions (proxy for humans, capable evaluators, scalability, cost-effectiveness) through a measurement-theory lens.
Problem
LLMs used as judges for NLG evaluation are adopted rapidly, but their validity and reliability as measurement instruments remain underexplored.
Approach
Applies measurement theory (construct validity and reliability, seven validity dimensions) from the social sciences to critically examine four implicit assumptions behind LLM-as-judge practice, illustrated with summarization, data annotation, and safety alignment case studies.
Key results
- Literature review shows SummEval fluency instructions are inconsistently adopted across LLJ papers, with varying scales (1-3, 1-5, 1-10, 1-100) and comparison types (absolute, pairwise, relative).
- Argues LLJs' validation via convergent validity with human judgments is undermined because human judgments themselves lack standardized collection practices.
- Reports that LLM safety judges can be made to misclassify up to 100% of harmful generations as harmless via simple modifications, as cited from prior work.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply simple prompt variations to safety judges | LLM safety judges · Safety classification | harmful generations misclassified as harmless: up to 100% | Hurt |
| Generate rationales through chain-of-thought reasoning | LLM judges · NLG evaluation | correlation with human judgments | Helped |
| Evaluate under high human uncertainty | LLM-based automatic metrics · Subjective annotation tasks | correlation with human judgments | Mixed |
| Add an apology to an unsafe prompt | LLM safety judges · Safety classification | harmful prompt classified as harmless | Hurt |
| Use a judge from a model family similar to the evaluated model | LLJ-based benchmarks · AlpacaEval and Arena-Hard | evaluation score bias | Hurt |
| Adapt responses to align more closely with the judge | Models evaluated by preference-based LLJs · Preference-based evaluation | evaluation score validity | Hurt |
| Use LLM judges in place of human annotators for subjective tasks | LLMs as annotators · Subjective and contested classification tasks | diversity of annotation judgments | Hurt |
| Use larger models or juries of smaller LLMs | LLJ evaluation · NLG evaluation | evaluation accuracy | Helped |
Surprising findings
- Cost savings of LLJs may be overstated: LLM-based evaluation can cost more than human annotation in some scenarios such as Arena-Hard, and AMT data quality has declined over time.
Limitations
- Qualitative, non-exhaustive literature review rather than systematic survey
- Relies largely on cited prior findings without new experiments
- Focus on NLG and high-level perspective may not cover all judge-specific biases
Experimental setup
Datasets: SummEval RealSumm Topical-Chat
Builds on: measurement theory Superficial Alignment Hypothesis
Closest papers by meaning
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.92)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.91)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.91)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.91)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.91)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.