Research map / Large language models / Language, safety and interpretability

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Soft-Elo, which trains Bradley-Terry on calibrated soft win probabilities from LLM judge score differences and wraps Elo estimates in split conformal intervals, cutting LLM-to-human Elo MAE from 45.9 to 17.9 on LMArena.

Problem

LLM-as-a-judge Elo rankings are miscalibrated relative to human Elo because judges carry systematic biases and hard win/tie/loss labels discard score-difference uncertainty, yielding wide and unreliable intervals.

Approach

Keep the Bradley-Terry model fixed but replace hard battle labels with soft targets sigma(beta*s(x)), where beta is fit by MLE on human non-tie battles; then apply split conformal prediction with bootstrap-normalized scores to LLM-human Elo residuals across models.

Key results

• Soft-Elo reduces mean held-out Elo MAE from 45.9 to 17.9 across eight judges on LMArena 100K, a 39-73% reduction per judge.
• Soft-Elo reduces median split-conformal width by 39-70% on LMArena 100K while keeping coverage near 90% nominal (92.1-96.4% vs Hard-Elo 88.6-97.9%).
• On LMArena 140K, Soft-Elo narrows conformal intervals by 30-73% (mean 56%); on ComparIA the reductions are 19-78%, smallest for Qwen judges.

Surprising findings

• Soft-Elo's advantage weakens on the French ComparIA corpus for Qwen judges, where fitted beta falls to 0.13-0.14 and Spearman rho drops substantially (e.g., Qwen3.5-27B 0.845 to 0.660).

Limitations

• Uses only uncertainty in judge score difference; does not model epistemic uncertainty, hallucinated scores, or judge instability.
• Conformal guarantee is marginal and relies on exchangeability between calibration and future models; shifts can violate it.
• Soft-Elo weakens when the score difference does not predict human agreement (ComparIA with Qwen judges).

Experimental setup

Datasets: LMArena 100K LMArena 140K ComparIA

Benchmarks: LMArena 100K LMArena 140K ComparIA

Metrics: Elo MAE Spearman rho Cohen's kappa conformal interval width empirical coverage

Baselines: Hard-Elo Bridge

Builds on: Bradley-Terry model Chatbot Arena split conformal prediction Bridge

Closest papers by meaning

• Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.92)
• Elo Uncovered: Robustness and Best Practices in Language Model Evaluation (NeurIPS 2024 · similarity 0.90)
• BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments (NeurIPS 2026 · similarity 0.90)
• Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.89)
• Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations (NeurIPS 2025 · similarity 0.89)
• JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.89)