Research map / Large language models / Language, safety and interpretability
Elo Uncovered: Robustness and Best Practices in Language Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether the Elo rating system is reliable for ranking LLMs from pairwise human feedback, testing transitivity and sensitivity to match ordering and K-factor using synthetic Bernoulli data and real human feedback on Dolly and Flan models.
Problem
Elo ratings are increasingly used to rank LLMs from "A vs B" human comparisons, but LLMs have static skill and matches can be run in any order, so it is unclear whether Elo rankings are robust and transitive.
Approach
Frames Elo evaluation through two axioms (transitivity and reliability to match ordering and K-factor), simulates pairwise outcomes as Bernoulli/binomial processes, and averages Elo over many random permutations of the same match outcomes to measure stability.
Key results
- Elo rankings are unstable for a single sequence when win probability is near 0.5 (e.g., 0.51); averaging over at least 100 permutations stabilizes ratings.
- In transitivity scenarios with models of similar performance, Elo rankings violated the expected hierarchy (starred cases in the table), especially at N=100 and K=1 for near-tied models.
- On real human feedback, Flan-T5-xxl vs Flan-T5-xl (win rate 0.64 vs 0.36) gave stable Elo across K and permutations, while the near-tied Dolly-v2-7b vs Dolly-v2-12b (0.51 vs 0.49) was volatile at N_perms=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Average ratings over 100 comparison-order permutations rather than one sequence | Elo ratings for similarly performing models · Synthetic pairwise comparisons, especially P(A_win) ≈ 0.5 | ranking stability | Helped |
| Use a higher K-factor with at least 100 permutations | Elo ratings for pairwise model comparisons · Synthetic comparisons with distinguishable win rates | convergence to model skill | Helped |
| Use a higher K-factor for a single comparison sequence | Elo ratings for closely matched models · Synthetic pairwise comparisons with winning probabilities nearing 0.5 | ranking stability | Hurt |
| Use K=1 instead of K=16 with 100 permutations | Elo transitivity rankings · A ≫ B, B ≈ C | ranking correctness: 1342.70* ± 0.53; 1361.38* ± 0.38 -> 1340.83 ± 2.83; 1333.13 ± 2.68 | Hurt |
| Use K=1 instead of K=16 with 100 permutations | Elo transitivity rankings · A ≈ B, B ≫ C | ranking correctness: 1433.84* ± 0.41; 1453.84* ± 0.61 -> 1460.22 ± 2.90; 1452.87 ± 3.25 | Hurt |
| Use 100 permutations instead of one | Elo ratings for human feedback comparisons · Dolly-v2-7b vs Dolly-v2-12b | rating stability | Helped |
| Use different K-factors and permutation counts | Elo ratings for human feedback comparisons · Flan-T5-xxl vs Flan-T5-xl | rating stability | No clear effect |
| Use different K-factors and permutation counts | Elo ratings for human feedback comparisons · Dolly-v2-7b vs Dolly-v2-12b | ranking stability | Mixed |
| Infer model ordering without comprehensive pairwise comparisons | Elo ratings for LLMs · Real-world human evaluations | transitivity | Hurt |
| Use a single comparison sequence rather than multiple permutations | Elo ratings for models with similar win rates · Synthetic and human-feedback comparisons | ranking stability | Hurt |
Surprising findings
- Transitivity of Elo rankings is not reliably guaranteed even with consistent pairwise outcomes, particularly for models with similar performance.
- Higher K-factors speed convergence for clear winners but amplify instability for single-sequence rankings of near-tied models.
Limitations
- Validation on real human feedback uses only a small set of models (Dolly and Flan families) and 400 prompts.
- Ties are excluded from the analysis; the multi-outcome extension is only in the appendix.
- Synthetic experiments rely on i.i.d. Bernoulli outcomes that may not capture real human feedback dynamics.
Experimental setup
Datasets: SODA CommonsenseQA CommonGen AdversarialQA P3
Benchmarks: synthetic Bernoulli pairwise outcomes human pairwise feedback on 400 prompts (SODA, CommonsenseQA, CommonGen, AdversarialQA)
Metrics: Elo score score difference between models ranking consistency across permutations and K-factors
Builds on: Elo rating system Glicko TrueSkill
Closest papers by meaning
- Re-evaluating Open-ended Evaluation of Large Language Models (ICLR 2025 · similarity 0.91)
- am-ELO: A Stable Framework for Arena-based LLM Evaluation (ICML 2025 · similarity 0.91)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.90)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.89)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.89)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.