Research map / Large language models / Language, safety and interpretability
Efficient Inference for Noisy LLM-as-a-Judge Evaluation
Yiqun T. Chen, Sizhu Lu, Sijia Li, Moran Guo, Shengyi Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EIF/PPI++ instead of standard PPI | PPI · Binary simulation (q0=q1) | CI width: 35--55% narrower | Improved |
| use PPI/EIF instead of Rogan-Gladen | Rogan-Gladen · Binary simulation (q0=q1) | CI width: 3--15x narrower | Improved |
| add debiasing (RG, PPI, PPI++, EIF) to naive judge mean | naive LLM-judge mean · Binary simulation | coverage of 90% CI | Improved |
| use Rogan-Gladen with small calibration set and weak judge | Rogan-Gladen · Binary simulation (small calibration, low judge quality) | coverage of 90% CI | Hurt |
| use flexible model (GAM/spline/per-class) for E[Y|Yhat] instead of linear | EIF-linear / PPI++ · Discrete-prediction simulation (mu3 varied) | CI width | Improved |
| use EIF-linear instead of PPI identity map | PPI · Discrete-prediction simulation (mu3 varied) | CI width | Improved |
| use EIF/PPI++/MLE instead of Rogan-Gladen | Rogan-Gladen · Chatbot Arena win rate, 1000 splits | CI width: 0.76--0.96 -> 0.27--0.30 | Improved |
| use PPI++/EIF instead of PPI | PPI · Chatbot Arena win rate, 1000 splits | CI width: 0.35--0.44 -> 0.27--0.30 | Improved |
| replace naive judge mean with debiased estimators | naive LLM-judge mean · Chatbot Arena win rate, 1000 splits | coverage of 90% CI: 0% coverage under substantial bias | Improved |
| use labeling budget of only 1% with direct q estimation | Rogan-Gladen · Binary simulation, 1% labeling | RMSE of q1: 0.3 vs 0.1 for q0 | Hurt |
Closest papers by meaning
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.91)
- Tuning LLM Judge Design Decisions for 1/1000 of the Cost (ICML 2025 · similarity 0.91)
- Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025 · similarity 0.91)
- Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees (NeurIPS 2025 · similarity 0.91)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.90)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
