Research map / Large language models / Language, safety and interpretability
A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| weight judges by learned reliability gamma instead of unweighted BTL, confidence interval width | Judge-aware ranking · In-house 45 models, 18 judges | average CI width: 0.211 -> 0.185 | Improved |
| weight judges by learned reliability, CI width on UltraFeedback | Judge-aware ranking · UltraFeedback (20 judges) | average CI width: 0.278 -> 0.264 | Improved |
| account for judge heterogeneity, Pearson agreement with reference | Judge-aware ranking · Mixed-quality judge pool | Pearson correlation: 0.8992 -> 0.9394 | Improved |
| account for judge heterogeneity, Spearman agreement with reference | Judge-aware ranking · Mixed-quality judge pool | Spearman correlation: 0.8316 -> 0.9212 | Improved |
| use judge-aware weighting, agreement with human ranking on Chatbot Arena | Judge-aware ranking · Chatbot Arena | Spearman correlation: 0.9699 -> 0.9955 | Improved |
| use judge-aware weighting, GPT-4 rank on Chatbot Arena | Judge-aware ranking · Chatbot Arena | GPT-4 rank: 3 -> 1 | Improved |
| increase judge budget K | Judge-aware vs unweighted · In-house subsampling | correlation gap between weighted and unweighted | Improved |
| weighted aggregation vs unweighted at small sample sizes | Judge-aware ranking · In-house subsampling | correlation to full-data reference | Improved |
| ignore judge heterogeneity (standard BTL) in simulation, CI coverage (ablation) | BTL · Simulation | 95% CI coverage | Hurt |
| remove tied comparisons (ablation) | Judge-aware ranking · MT-Bench/Arena | ranking and CIs | No clear effect |
Closest papers by meaning
- Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence (NeurIPS 2026 · similarity 0.93)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.91)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.91)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.90)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.90)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
