Research map / Large language models / Alignment and preferences
Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes selective LLM-as-judge evaluation with a provable human-agreement guarantee via fixed sequence testing, a new in-context "Simulated Annotators" confidence measure, and a cascade of cheaper-to-stronger judges.
Problem
LLM-based pairwise evaluation lacks a reliable guarantee of agreement with human judgments, and standard confidence measures such as predictive probability and verbalized confidence are over-confident, limiting coverage when abstaining.
Approach
Selective evaluation with an abstention threshold calibrated by fixed sequence testing on a small human-labeled calibration set gives a high-probability guarantee on disagreement risk; Simulated Annotators estimates confidence as agreement across few-shot-prompted simulated annotators; a cascade of judges (Mistral-7B, GPT-3.5, GPT-4) escalates only uncertain instances.
Key results
- On ChatArena with target 80% agreement, Cascaded Selective Evaluation covers 79.1% of samples with 88.1% of those evaluated by Mistral-7B or GPT-3.5 (abstract/intro).
- On TL;DR at target 0.9, the method achieves 90.8% guarantee success rate with 55.7% coverage, while point-estimate calibration reaches 47.5-79.0%.
- Simulated Annotators reduces ECE by 50% and improves AUROC by 13% over predictive probability for GPT-4-turbo on AlpacaEval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cascaded selective evaluation instead of heuristic selection | LLM judges · TL;DR | Guarantee Success Rate (%): 42.0 -> 90.8 | Helped |
| Use cascaded selective evaluation instead of GPT-4 point-estimate calibration | LLM judges · ChatArena | Guarantee Success Rate (%): 54.4 -> 91.0 | Helped |
| Use simulated annotator confidence instead of predictive probability | GPT-4-turbo judge · AlpacaEval | ECE: 0.217 -> 0.095 | Helped |
| Use simulated annotator confidence instead of predictive probability | GPT-4-turbo judge · TL;DR | ECE: 0.196 -> 0.039 | Helped |
| Randomize labels in annotator examples instead of simulating individual preferences (ablation) | GPT-4-turbo confidence estimation · AlpacaEval | ECE: 0.095 -> 0.113 | Hurt |
| Use majority human labels instead of individual annotator labels (ablation) | GPT-4-turbo confidence estimation · AlpacaEval | ECE: 0.095 -> 0.106 | Hurt |
| Increase simulated annotators from 1 to 5 (ablation) | Cascaded Selective Evaluation · ChatArena | Coverage (%): 60.9 -> 63.2 | Helped |
| Replace GPT-4 with Mixtral-8×7B in the judge cascade (ablation) | Cascaded Selective Evaluation · ChatArena | Relative API Cost: 0.215 -> 0.126 | Helped |
| Replace GPT-4 with Mixtral-8×7B in the judge cascade (ablation) | Cascaded Selective Evaluation · ChatArena | Guarantee Success Rate (%): 90.5 -> 90.8 | No clear effect |
Surprising findings
- Predictive probability and verbalized confidence overestimate human agreement even for GPT-4-turbo, the strongest judge.
- Weaker cascades (Mistral-7B, Mixtral, GPT-3.5) still meet the target agreement while cutting cost to 12.6% of GPT-4 evaluation.
Limitations
- Guarantee relies on an i.i.d. calibration set; performance degrades slightly under distribution shift (coverage drops).
- Human annotation of calibration data is required, and only one human annotation per instance for some datasets (majority-based variant).
Experimental setup
Datasets: TL;DR Chatbot Arena AlpacaEval Auto-J
Benchmarks: TL;DR ChatArena Auto-J AlpacaEval
Metrics: empirical human agreement coverage guarantee success rate ECE AUROC AUPRC relative API cost
Baselines: Predictive Probability Verbalized Confidence Randomized Annotators Heuristic Selection Cascaded Heuristic Selection Point-Estimate Calibration Semantic Entropy
Builds on: conformal risk control / selection with guaranteed risk fixed sequence testing semantic entropy
Closest papers by meaning
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.89)
- Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025 · similarity 0.89)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.88)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.88)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.88)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.