Research map / Large language models / AI and society
Validating LLM-as-a-Judge Systems under Rating Indeterminacy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a framework for validating LLM-as-a-judge systems under rating indeterminacy, showing forced-choice elicitation biases judge selection and recommending multi-label response-set ratings with MSE-based agreement.
Problem
Validating LLM-as-a-judge systems when items admit multiple valid ratings (rating indeterminacy), where forced-choice human ratings hide intra-rater ambiguity and can mis-select judge systems.
Approach
Model how raters resolve indeterminacy via a forced-choice translation matrix linking multi-label response-set ratings to forced-choice ratings; use multi-label (response set) elicitation and continuous multi-label MSE to measure human-judge agreement, rather than forced-choice metrics like Hit Rate or KL divergence.
Key results
- Forced-choice-selected top judge (Claude Sonnet 3.5) performs 31% worse than the true best judge (GPT o3-Mini) on the toxicity task under downstream metrics.
- Distributional KL-D(h,j) performs best among forced-choice metrics, but continuous multi-label MSE selects more performant judges when beta^H >= 0.2.
- Estimated sensitivity parameters for SummEval (Relevance) range from 0.12 to 0.54 across judge systems, showing judges resolve indeterminacy differently.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace forced-choice validation with response set ratings for judge selection | LLM-as-a-judge validation on Civil Comments toxicity · Civil Comments (toxicity) | decision consistency: 31\% worse | Helped |
| select the judge using KL-D(j,h) rather than the downstream metric | LLM-as-a-judge validation · SummEval (Relevance) | estimation bias: 28\%; 0.28 | Hurt |
| use response set-based MSE rather than forced-choice agreement metrics when rating indeterminacy is present | LLM-as-a-judge validation · 11 rating tasks | downstream performance regret | Helped |
| estimate the forced-choice translation matrix using a small paired-rating subset | MSE F validation · 11 rating tasks | downstream performance regret | Helped |
| use KL-D(h,j) rather than other forced-choice metrics | LLM-as-a-judge validation · 11 rating tasks | downstream performance regret | Helped |
| use KL-D(h,j) instead of KL-D(j,h) | LLM-as-a-judge validation · 11 rating tasks | downstream performance regret | Helped |
Surprising findings
- The true best-performing judge is ranked fourth under the standard forced-choice validation approach.
- KL divergence measured in the (human, judge) direction outperforms the (judge, human) direction among forced-choice metrics.
Limitations
- Framework only covers closed-form discrete-option rating tasks, not continuous scales.
- Rating model assumes homogeneity of the human rating process across raters, which may be violated.
- Response-set elicitation may increase cognitive load and rating cost for human raters.
Experimental setup
Datasets: JudgeBench SNLI MNLI α-NLI SummEval QAGS Civil Comments
Benchmarks: SNLI MNLI α-NLI SummEval QAGS Civil Comments TopicalChat
Metrics: Hit Rate Cohen's kappa KL divergence MSE decision consistency estimation bias regret
Baselines: Hit Rate Cohen's kappa Krippendorff's alpha Fleiss' kappa KL divergence (forced-choice) JS divergence Cross-entropy Coverage
Builds on: perspectivism in NLP KL divergence agreement metrics
Closest papers by meaning
- Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions (AAAI 2025 · similarity 0.92)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.91)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.90)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.90)
- Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations (NeurIPS 2025 · similarity 0.89)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.