Research map / Large language models / Alignment and preferences
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Chatbot Arena, a crowdsourced live platform where users compare anonymous LLM pairs and vote, and analyzes the resulting data with Bradley-Terry ranking, adaptive sampling, and expert agreement checks.
Problem
Static, ground-truth benchmarks fail to capture open-ended, real-world human preferences and risk contamination, so LLMs need a live, scalable evaluation based on human preference.
Approach
A free, open website where users submit any prompt to two anonymous randomly sampled LLMs and vote for the better response; votes feed a Bradley-Terry ranking with sandwich confidence intervals and an active pair-sampling rule that reduces the data needed for precise rankings.
Key results
- Adaptive sampling reduced samples needed: 4,400 vs 6,800 for theta precision 0.2 (random needs 54% more data).
- Crowd-vs-expert agreement ranged 72.8% to 83.1%, comparable to expert-vs-expert agreement of 79.4% to 89.8%.
- Anomalous-user detector achieved about 90% true positive rate with 60-70% true negative rate at alpha 0.1 in a small test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive rather than random model-pair sampling | Chatbot Arena ranking estimation · Win matrix estimation | samples to precision 0.2: 6,800 -> 4,400; random baseline requires 54% more data | Helped |
| Use adaptive rather than random model-pair sampling | Chatbot Arena ranking estimation · Score estimation | samples to precision 0.3: 17,200 -> 16,400; random baseline requires 5% more data | Helped |
| Evaluate topic-cluster prompts requiring coding and reasoning | GPT-4-0613 vs Llama-2-70b-chat · Python Game Programming Challenge | win rate: 96.7% | Helped |
| Evaluate topic-cluster prompts with less problem-solving | GPT-4-0613 vs Llama-2-70b-chat · Movie Recommendations & Ratings | win rate: 53.3% | Mixed |
| Collect crowd-user pairwise votes instead of expert judgments | Chatbot Arena vote evaluation · GPT-4-Turbo vs Llama-2-13B | agreement rate: 72.8%; 77.8% | Mixed |
Surprising findings
- Expert-vs-expert agreement was only about 79-90%, showing that many prompts lack a single ground-truth answer.
- GPT-4 judge agreement with crowd and experts was similar to expert-expert agreement levels in several settings.
Limitations
- User base skews toward LLM hobbyists and researchers, possibly biasing the prompt distribution.
- Data comes from the online chat interface and may not reflect production or specialized-domain use.
- Focuses on helpfulness and does not evaluate safety.
Experimental setup
Datasets: Chatbot Arena crowdsourced pairwise votes (about 240K votes, 213,576 used in replay)
Benchmarks: Chatbot Arena votes Arena Bench MT-Bench
Metrics: win-rate agreement rate Bradley-Terry coefficient confidence interval coverage sample efficiency
Baselines: MT-Bench Elo rating random sampling
Builds on: Bradley-Terry model BERTopic MT-Bench Elo rating
Closest papers by meaning
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.93)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.92)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.92)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.91)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.90)
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.