Research map / Large language models / Language, safety and interpretability
am-ELO: A Stable Framework for Arena-based LLM Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces iterative ELO updates with maximum likelihood estimation (m-ELO) and adds per-annotator ability parameters (am-ELO) to yield a more stable arena ranking framework.
Problem
Arena-based LLM evaluation with ELO is unstable because scores depend on battle record order, and annotator reliability differences are ignored.
Approach
Recast ELO as maximum likelihood estimation over all records (m-ELO) and replace the fixed scale constant with a per-annotator ability theta_k estimated jointly with model scores, normalized to sum to 1 so that negative-ability annotators are flagged as anomalous.
Key results
- am-ELO achieved MSE 0.1208 and AUC 0.7581 on prediction, vs ELO 0.1238/0.7492 and m-ELO 0.1234/0.7503.
- Under simulated annotator perturbations, am-ELO reduced ranking inconsistency to about 30% of ELO/m-ELO levels in three of four perturbation settings.
- Anomalous-annotator detection reached about 90% F1 at threshold 0 and 95% at threshold 0.005.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace iterative ELO updates with MLE | ELO · Chatbot | MSE: 0.1238$\pm$ 0.0031 -> 0.1234$\pm$ 0.0029 | No clear effect |
| Replace iterative ELO updates with MLE | ELO · Chatbot | AUC: 0.7492$\pm$ 0.0068 -> 0.7503$\pm$ 0.0066 | No clear effect |
| Add annotator-ability modeling to the ELO probability function | m-ELO · Chatbot | MSE: 0.1234$\pm$ 0.0029 -> \textbf{0.1208}$\pm$ 0.0034 | Helped |
| Add annotator-ability modeling to the ELO probability function | m-ELO · Chatbot | AUC: 0.7503$\pm$ 0.0066 -> \textbf{0.7581}$\pm$ 0.0067 | Helped |
| Remove annotator-ability normalization (ablation) | am-ELO · Chatbot | ranking consistency: 0.4 | Hurt |
| Use annotator-ability modeling instead of m-ELO or traditional ELO | am-ELO · Chatbot, Random/Flip/Mixed perturbations | ranking inconsistency rate: 30% | Helped |
| Detect anomalous annotators using estimated ability with threshold ε=0 | am-ELO · Simulated annotator perturbations | F1-score: 90% | Helped |
| Detect anomalous annotators using estimated ability with threshold ε=0.005 | am-ELO · Simulated annotator perturbations | F1-score: 95% | Helped |
Surprising findings
- The unnormalized am-ELO variant converged fastest but reached different local minima across random initializations, with consistency stabilizing at 0.4.
Limitations
- Annotator modeling captures only discriminative ability and consistency with other annotators, a simplistic view of annotator capability.
- Stability evaluated mainly via simulated perturbations rather than live arena deployment.
- Filtering annotators with fewer than 50 records limits the dataset and the method's applicability to sparse arenas.
Experimental setup
Datasets: Chatbot Arena dataset (Chatbot)
Benchmarks: Chatbot Arena dataset
Metrics: MSE AUC log-likelihood loss pairwise ranking consistency F1-score for anomalous annotator detection
Baselines: ELO m-ELO
Builds on: ELO rating system Bradley-Terry model Item Response Theory (2PL)
Closest papers by meaning
- Elo Uncovered: Robustness and Best Practices in Language Model Evaluation (NeurIPS 2024 · similarity 0.91)
- Re-evaluating Open-ended Evaluation of Large Language Models (ICLR 2025 · similarity 0.89)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.88)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.86)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.86)
- Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.