Research map / Label-efficient and robust learning / Recognition and long-tail
Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that aggregate correlation between LLM-as-a-judge and human labels can mislead under human label uncertainty, and proposes stratifying by human uncertainty, a binned Jensen-Shannon divergence metric, and perception charts.
Problem
Automatic evaluation (including LLM-as-a-judge) is validated by correlation with human labels, but chance-adjusted inter-rater reliability metrics assume human raters are interchangeable and that single gold labels exist, which breaks down under human label variation and perception-based judgments.
Approach
Stratify human-machine agreement by human label uncertainty (percentage agreement and unique label counts), replace single-gold-label correlation with a binned Jensen-Shannon divergence over full human label distributions, and visualize with perception charts.
Key results
- On MNLI matched, when all human labels agree (PA=1, 58% of samples), HH Krippendorff's alpha is 1.00 versus 0.83 for human-machine majority, a gap of 0.17 that widens to larger differences as human consistency increases.
- Under high human uncertainty (unique=3), HM correlation exceeds HH (e.g., alpha -0.45 difference favoring machines on SNLI), creating the illusion that LJs approximate human majority.
- A random labeler appears to have similar or better correlation with synthetic human labels when uncertainty is high, but performs poorly when human labels are consistent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Stratify MNLI matched examples by human agreement, comparing machine-majority agreement in the perfectly agreed subset with the low-agreement subset | Sonnet 3 human-majority to machine-majority evaluation · MNLI matched | Krippendorff's alpha: 0.83 vs. 1.00; 0.37 vs. 0.04 | Mixed |
| Stratify SNLI examples by human agreement | Sonnet 3 human-majority to machine-majority evaluation · SNLI | Krippendorff's alpha: All: 0.63 vs. 0.68; PA = 1: 0.69 vs. 1.00; 0 ≤ PA < 0.8: 0.37 vs. 0.03 | Mixed |
| Stratify SummEval coherence by perfect human agreement | Mistral Large v2 human-median to machine-median evaluation · SummEval coherence | Krippendorff's alpha: All: 0.49 vs. 0.55; PA = 1: 0.35 vs. 1.00; PA < 60%: 0.55 vs. 0.19 | Mixed |
| Use human mean instead of median aggregation | SummEval coherence human-machine comparison · SummEval coherence | Spearman's rho: 0.49 → 0.60 | Helped |
| Compare full human and machine label distributions using binned JSD rather than a single gold label | Good and poor models in the toy Likert example · Toy Likert example | Binned Jensen-Shannon divergence: 0.56 vs. 0.65 | Helped |
| Compare the toy example with Kendall's tau instead of binned JSD | Good and poor models · Toy Likert example | Kendall's tau: 0.00 vs. 0.82 | Hurt |
| Compare the toy example with Spearman's rho instead of binned JSD | Good and poor models · Toy Likert example | Spearman's rho: 0.00 vs. 0.87 | Hurt |
| Compare the toy example with Krippendorff's alpha instead of binned JSD | Good and poor models · Toy Likert example | Krippendorff's alpha: -0.06 vs. -0.06 | No clear effect |
Surprising findings
- A random labeler can match or exceed LLM judges' correlation with human majority labels when human label uncertainty is high.
- Rankings of LLM judges and absolute correlation numbers change depending on which human reference annotator is used under high uncertainty.
Limitations
- Prompts were reused from GPT-4 judge work and not optimized per LLM, so results do not rank the judges.
- Binned JSD is computed per bin rather than per item, so item-level discrepancies may be hidden.
- Limited number of datasets and human annotations per item (e.g., 3 to 5 labels).
Experimental setup
Datasets: SummEval SNLI MNLI MT-Bench JudgeBench
Benchmarks: SummEval SNLI MNLI MT-Bench
Metrics: Krippendorff's alpha percentage agreement Fleiss kappa Randolph's kappa Spearman's rho Kendall's tau binned Jensen-Shannon divergence
Baselines: G-Eval Krippendorff's alpha Spearman's rho Kendall's tau Fleiss's kappa Randolph's kappa Rouge-1
Builds on: Krippendorff's alpha Randolph's kappa G-Eval
Closest papers by meaning
- BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments (NeurIPS 2026 · similarity 0.88)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.88)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.87)
- Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations (NeurIPS 2025 · similarity 0.86)
- Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models (AAAI 2025 · similarity 0.85)
- Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.