AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou

ICML 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
weight judges by learned reliability gamma instead of unweighted BTL, confidence interval widthJudge-aware ranking · In-house 45 models, 18 judgesaverage CI width: 0.211 -> 0.185Improved
weight judges by learned reliability, CI width on UltraFeedbackJudge-aware ranking · UltraFeedback (20 judges)average CI width: 0.278 -> 0.264Improved
account for judge heterogeneity, Pearson agreement with referenceJudge-aware ranking · Mixed-quality judge poolPearson correlation: 0.8992 -> 0.9394Improved
account for judge heterogeneity, Spearman agreement with referenceJudge-aware ranking · Mixed-quality judge poolSpearman correlation: 0.8316 -> 0.9212Improved
use judge-aware weighting, agreement with human ranking on Chatbot ArenaJudge-aware ranking · Chatbot ArenaSpearman correlation: 0.9699 -> 0.9955Improved
use judge-aware weighting, GPT-4 rank on Chatbot ArenaJudge-aware ranking · Chatbot ArenaGPT-4 rank: 3 -> 1Improved
increase judge budget KJudge-aware vs unweighted · In-house subsamplingcorrelation gap between weighted and unweightedImproved
weighted aggregation vs unweighted at small sample sizesJudge-aware ranking · In-house subsamplingcorrelation to full-data referenceImproved
ignore judge heterogeneity (standard BTL) in simulation, CI coverage (ablation)BTL · Simulation95% CI coverageHurt
remove tied comparisons (ablation)Judge-aware ranking · MT-Bench/Arenaranking and CIsNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy