Research map / Large language models / AI and society
A Statistical Framework for Ranking LLM-based Chatbots
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a statistical framework for ranking LLM chatbots from pairwise human votes, adding a factored tie model, Thurstonian covariance modeling, and identifiability constraints, released as the leaderbot package.
Problem
Elo-style rating used by Chatbot Arena ignores ties and inter-competitor covariance, and paired-comparison models for chatbot votes suffer from poor tie fit and parameter non-uniqueness during optimization.
Approach
Generalizes Rao-Kupper and Davidson tie models with a low-rank additive factor model for pair-specific tie thresholds (DCT-IV basis), combines them with Thurstonian covariance factor models, and adds constraints that remove shift and scale symmetries and a translation symmetry in the factor loadings.
Key results
- Generalized factored tie models reduce tie-fitting errors by about two orders of magnitude relative to the original Rao-Kupper and Davidson models, per the paper's claim.
- Among 30 configurations, models with Thurstonian covariance form a distinct cluster in Kendall-tau hierarchical clustering, indicating covariance is the main driver of ranking differences.
- Top two chatbots in the ranking are ChatGPT-4o Latest and Gemini 1.5 Pro Experimental, forming Tier I, with hierarchical clustering recovering rank order.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pair-specific factored tie parameters instead of a single tie parameter | Generalized Rao-Kupper and Davidson models · Chatbot Arena | prediction error: two orders of magnitude | Helped |
| Use pair-specific factored tie parameters instead of a single tie parameter | Generalized Rao-Kupper and Davidson models · Chatbot Arena | win/loss prediction error: two orders of magnitude | Helped |
| Use additive rather than multiplicative structure for pairwise tie parameters | Factored tie model · Chatbot Arena | tie modeling effectiveness | Helped |
| Use a single tie parameter rather than factored tie parameters (ablation) | Rao-Kupper model · Chatbot Arena | tie prediction accuracy | Hurt |
| Include Thurstonian covariance factors | Paired-comparison models · Chatbot Arena | ranking similarity | Mixed |
| Add constraints to eliminate score-shift, covariance-scaling, and factor-translation symmetries | Thurstonian paired-comparison models · Chatbot Arena | optimization stability | Helped |
| Use the original Bradley-Terry model without ties rather than a tie-aware model (ablation) | Bradley-Terry model · Chatbot Arena | win/loss matrix accuracy | Hurt |
Surprising findings
- Covariance inclusion, rather than tie modeling, is the primary driver of ranking differences across models, more so than tie parameters.
Limitations
- Ties are modeled only through the chosen parametric families (Rao-Kupper, Davidson), so fit depends on these forms.
- Score parameters are unidentifiable individually; only differences are meaningful, limiting uncertainty quantification for individual scores.
Experimental setup
Datasets: Chatbot Arena pairwise comparison data
Benchmarks: Chatbot Arena pairwise win/loss/tie data
Metrics: fit error on win/loss/tie matrices Kendall rank correlation between model rankings prediction error
Baselines: Bradley-Terry with ties as half win/loss Bradley-Terry without ties original Rao-Kupper model original Davidson model
Builds on: Bradley-Terry model Rao-Kupper model Davidson model Thurstonian model Elo rating system
Closest papers by meaning
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.93)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.92)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.90)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.90)
- Prediction-Powered Ranking of Large Language Models (NeurIPS 2024 · similarity 0.90)
- Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.