Research map / Large language models / Language, safety and interpretability
MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MixEval, a ground-truth LLM benchmark made by matching web-mined user queries (detected with a fine-tuned Vicuna 33B on Common Crawl) to similar queries in existing benchmarks, plus a harder subset MixEval-Hard and a fast dynamic update mechanism.
Problem
LLM benchmarks are biased in query distribution, grading (LLM judges), and generalization (contamination), while user-facing evaluation like Chatbot Arena is costly, slow, and irreproducible.
Approach
Mine real-world user queries from Common Crawl with a trained query detector, then map each web query to its most similar query in a pool of ground-truth benchmarks and use that benchmark item as the evaluation item, yielding a ground-truth benchmark whose query distribution mirrors real users; a difficulty-weighted rejection sampler yields MixEval-Hard.
Key results
- MixEval and MixEval-Hard reach 0.93 and 0.96 Spearman correlation with Chatbot Arena Elo, versus roughly 10% lower for the best individual benchmarks.
- MixEval costs about 6% of the time and cost of MMLU, and its data updates complete within one minute.
- Across five MixEval versions, five models show a score Std. of 0.36 and an 85.05% unique benchmark query ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mix web-mined queries with similar ground-truth benchmark queries | Individual ground-truth benchmarks · Arena Elo | correlation | Helped |
| Use the MixEval sample-level mixture instead of benchmark-level or uniform mixture (ablation) | Benchmark mixture · Arena Elo | correlation | Helped |
| Select a difficulty-prioritized subset while maintaining a balanced query distribution (ablation) | MixEval · Chatbot Arena | ranking correlation: 0.93 -> 0.96 | Helped |
| Update benchmark versions by sampling different batches of web queries | MixEval · Five updated versions | score standard deviation: 0.36 | No clear effect |
Surprising findings
- Chat and base model scores of the same series correlate at 0.95, suggesting pre-training capability drives human preference more than post-training.
- GSM8K, despite skewed query distribution, achieves a 0.78 correlation with human preference, while BoolQ, with high topic overlap, achieves only 0.37.
Limitations
- The web query detector is not perfectly accurate and Common Crawl introduces biases, so the benchmark is not fully unbiased.
- Only text-in-text-out queries are covered; other I/O modalities are left to future work.
- Free-form grading relies on GPT-3.5-Turbo as a model parser, and the open-source parser is not yet provided.
Experimental setup
Datasets: Common Crawl MMLU BoolQ HellaSwag ARC CommonSenseQA AGIEval OpenbookQA GPQA WinoGrande TriviaQA DROP
Benchmarks: MixEval MixEval-Hard Chatbot Arena
Metrics: Spearman rank correlation with Arena Elo accuracy score standard deviation across versions cost and time
Baselines: Chatbot Arena MMLU MT-Bench Arena-Hard AlpacaEval-2.0 WildBench
Builds on: Chatbot Arena MMLU MT-Bench Vicuna
Closest papers by meaning
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.92)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.91)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.90)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.