Research map / Large language models / Agents, code and math
LiveBench: A Challenging, Contamination-Limited LLM Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LiveBench, a contamination-limited LLM benchmark with frequently refreshed, objectively graded questions across six categories, and evaluates about 40 models.
Problem
LLM benchmarks become unreliable due to test set contamination, while LLM-judge and human-crowdsourced alternatives introduce biases and fail on hard questions.
Approach
Build a benchmark whose questions are drawn from recent information sources (new math competitions, arXiv abstracts, Guardian news, IMDb synopses, Kaggle/Socrata datasets) or are harder contamination-limited versions of existing tasks, scored automatically against ground truth with no LLM judge, and refreshed monthly.
Key results
- o1-preview-2024-09-12 achieves the top LiveBench score of 64.7, with no model above 70%.
- LiveBench correlates 0.91 with ChatBot Arena and 0.88 with Arena-Hard across common models.
- Rank correlation between original and first update, and first and second update, both exceed 0.997; median and mean scores dropped about 1.2% on the latest questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4-Turbo LLM judging instead of ground-truth scoring (ablation) | Scoring challenging math and reasoning answers · AMC12 2024, AIME 2024, SMC 2023, and Zebra Puzzles | error rate: 0.380, 0.214, 0.353, 0.420 | Hurt |
| Use GPT-4-Turbo LLM judging instead of ground-truth scoring (ablation) | Claude-3-Opus answers on challenging math and reasoning tasks · AMC12 2024, AIME 2024, SMC 2023, and Zebra Puzzles | error rate: 0.388, 0.103, 0.294, 0.460 | Hurt |
| Use GPT-4-Turbo LLM-judged scores instead of ground-truth scores (ablation) | GPT-4-Turbo answers · AMC12 2024 | score: 54 -> 64.000 | Mixed |
| Use GPT-4-Turbo LLM-judged scores instead of ground-truth scores (ablation) | GPT-4-Turbo answers · AIME 2024 | score: 13.793 -> 35.714 | Mixed |
| Use GPT-4-Turbo LLM-judged scores instead of ground-truth scores (ablation) | GPT-4-Turbo answers · SMC 2023 | score: 70.588 -> 58.824 | Mixed |
| Use GPT-4-Turbo LLM-judged scores instead of ground-truth scores (ablation) | GPT-4-Turbo answers · Zebra Puzzles | score: 38 -> 68 | Mixed |
| Use GPT-4-Turbo LLM-judged scores instead of ground-truth scores (ablation) | Claude-3-Opus answers · AMC12 2024 | score: 56 -> 42.857 | Mixed |
| Use GPT-4-Turbo LLM-judged scores instead of ground-truth scores (ablation) | Claude-3-Opus answers · AIME 2024 | score: 6.897 -> 17.241 | Mixed |
| Use GPT-4-Turbo LLM-judged scores instead of ground-truth scores (ablation) | Claude-3-Opus answers · SMC 2023 | score: 58.824 -> 52.941 | Mixed |
| Use GPT-4-Turbo LLM-judged scores instead of ground-truth scores (ablation) | Claude-3-Opus answers · Zebra Puzzles | score: 34 -> 52 | Mixed |
| Refresh benchmark questions monthly | LiveBench · LiveBench updates | mean and median average score: about 1.2% | Hurt |
Surprising findings
- LLM judges show error rates of 0.10 to 0.46 on hard AMC, AIME, SMC and Zebra puzzle grading, suggesting they are unreliable for challenging math and reasoning.
- GPT-4 models score substantially higher on Arena-Hard than on LiveBench, likely due to self-judging bias.
Limitations
- No non-English tasks yet.
- Ground-truth scoring cannot handle open-ended tasks such as writing a travel guide.
- Prompt-format biases favor certain LLM families; prompts are not yet updated.
Experimental setup
Datasets: AMC12 2023 SMC 2023 AIME 2024 USAMO 2024 IMO 2024 LiveCodeBench IFEval-style instructions The Guardian articles arXiv abstracts IMDb/Wikipedia movie synopses Kaggle and Socrata tables Big-Bench Hard Web of Lies
Benchmarks: LiveBench ChatBot Arena Arena-Hard
Metrics: accuracy category average score Pearson correlation rank correlation error rate of LLM judge
Baselines: ChatBot Arena Arena-Hard AlpacaEval MT-Bench Omni-MATH SEAL leaderboard
Builds on: LiveCodeBench IFEval Big-Bench Hard AMPS MT-Bench
Closest papers by meaning
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.90)
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.90)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.89)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.89)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.88)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.