Research map / Large language models / Alignment and preferences
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WildBench, a benchmark of 1,024 challenging real-user tasks from WildChat, scored by GPT-4-Turbo judges using task checklists with pairwise WB-Reward (three baselines, length-penalty ties) and individual WB-Score.
Problem
Automated, cost-effective evaluation of LLMs on open-ended, challenging tasks that reflect the natural distribution of real user queries, as existing benchmarks (AlpacaEval, MT-Bench, ArenaHard) are narrow or skewed in task composition.
Approach
Curate hard real-user tasks from WildChat, use instance-specific checklists and CoT-style structured LLM-as-judge evaluation, compare each model against three baselines of differing strength for pairwise WB-Reward, and mitigate length bias by converting slight wins/losses to ties when the length gap exceeds K characters.
Key results
- WB-Reward (Mix, infinite K) achieves Pearson correlation 0.984 with Chatbot Arena Hard-English Elo on top-6 models, versus 0.909 for Arena-Hard and 0.892 for AlpacaEval2-LC.
- WB-Score reaches Pearson 0.955 on top models versus 0.909 for Arena-Hard and 0.892 for AlpacaEval2-LC on the same comparison.
- Checklists raise WB-Score Pearson correlation from 0.905 to 0.925 across all models; K=500 length penalty gives the best correlation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add task-specific checklists to the WB-Score prompt (ablation) | WB-Score · Chatbot Arena Elo correlation, all models | Pearson correlation: 0.905 -> 0.925 | Helped |
| use a length-penalty threshold of K=500 (ablation) | WB-Reward · Chatbot Arena Elo correlation, top 6 models | Pearson correlation: 0.984 -> 0.992 | Helped |
| use multiple LLM judges instead of a single judge (ablation) | WildBench evaluation · model relative rankings | relative ranking | No clear effect |
| use Claude 3 Haiku as the WB-Reward baseline (ablation) | WB-Reward · Chatbot Arena Elo correlation | Pearson correlation: 0.985 | Helped |
| use WB-Score instead of WB-Reward | WildBench evaluation · WildBench V2 | evaluation cost and speed: 10 minutes, $5; 3-4 times the cost | Helped |
Surprising findings
- Llama-3-8B-Inst-SimPO ranks well above Llama-3-70B-Inst on AlpacaEval 2.0 but is below it on WildBench, which the authors attribute to task selection bias in AlpacaEval.
- Using multiple LLM judges (GPT-4, Claude 3 Opus, Mistral-Large) produced very similar rankings, so the authors recommend a single judge in practice.
Limitations
- Current version covers English only.
- Test data sent via APIs to closed models risks future leakage; a private WildChat subset is reserved but the benchmark relies on continual updates.
- Multi-turn history responses were generated by GPT-4 rather than the evaluated model, and data reflects WildChat user demographic bias.
Experimental setup
Datasets: WildChat ShareGPT ChatbotArena Elo (Hard-English, 2024-05-20)
Benchmarks: WildBench AlpacaEval 2.0 Arena-Hard MT-Bench Chatbot Arena Hard-English Elo
Metrics: WB-Reward WB-Score Pearson correlation Spearman correlation Kendall correlation
Baselines: AlpacaEval 2.0 Arena-Hard MT-Bench Chatbot Arena Elo
Builds on: Chain-of-Thought prompting AlpacaEval Arena-Hard WildChat
Closest papers by meaning
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.92)
- From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline (ICML 2025 · similarity 0.90)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.90)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.90)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.90)
- WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.