Research map / Large language models / Alignment and preferences
JudgeBench: A Benchmark for Evaluating LLM-Based Judges
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JudgeBench, a 350-pair benchmark built by sampling GPT-4o responses to hard knowledge, reasoning, math, and coding questions and pairing correct with incorrect ones, to test whether LLM judges and reward models can detect objective correctness.
Problem
Existing LLM-judge benchmarks rely on crowdsourced human preferences or instruction-following, which fail to measure factual and logical correctness on hard tasks where human annotators are unreliable.
Approach
Convert existing hard datasets with verifiers into response pairs by sampling k responses from one strong model and keeping questions with at least one correct and one incorrect response, so labels reflect objective correctness and style is controlled; evaluate judges with both presentation orders and aggregate decisions.
Key results
- Under the Arena-Hard prompt, GPT-4o as judge reaches 56.57% overall on JudgeBench, and the vanilla prompt gives 50.86%, near random.
- o3-mini (high) achieves the best overall judge accuracy of 80.86%, while Claude-3.5-Sonnet reaches 64.29%.
- Reward models range from 59% to 64% overall, with Skywork-Reward-Gemma-2-27B at 64.29%, comparable to Claude-3.5-Sonnet as an LLM judge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the Arena-Hard prompt instead of the Vanilla prompt | GPT-4o judge on JudgeBench · JudgeBench Overall | accuracy: 50% to 56% | Helped |
| fine-tune Llama-3.1-8B as a Skywork judge instead of prompting the base model | Llama-3.1-8B judge · JudgeBench Overall | accuracy: over 12% | Helped |
| fine-tune Llama-3.1-70B as a Skywork judge instead of prompting the base model | Llama-3.1-70B judge · JudgeBench Overall | accuracy: 5% | Helped |
| increase reasoning level from low to high | o3-mini Arena-Hard Judge · JudgeBench Overall | accuracy: 70.57% to 80.86% | Helped |
| use the Arena-Hard judge to verify candidate solutions rather than solve the problem directly (ablation) | GPT-4o · JudgeBench Overall | accuracy: 54.57 to 56.57 | Helped |
| use the Arena-Hard judge to verify candidate solutions rather than solve the problem directly (ablation) | Claude-3.5-Sonnet · JudgeBench Coding | accuracy: 88.10 to 64.29 | Hurt |
| use the Arena-Hard judge to verify candidate solutions rather than solve the problem directly (ablation) | GPT-4o · JudgeBench Coding | accuracy: 73.81 to 59.52 | Hurt |
| use the Arena-Hard judge to verify candidate solutions rather than solve the problem directly (ablation) | GPT-4o · JudgeBench Math | accuracy: 58.93 to 75.00 | Helped |
| generate JudgeBench pairs with Claude-3.5-Sonnet instead of GPT-4o (ablation) | Claude-3.5-Sonnet judge · JudgeBench Overall | accuracy: 64.3% to 44.8% | Hurt |
| generate JudgeBench pairs with Claude-3.5-Sonnet instead of GPT-4o (ablation) | GPT-4o judge · JudgeBench Overall | accuracy: 56.6% to 53.9% | Hurt |
| increase the knowledge subset from 154 to 770 response pairs (ablation) | JudgeBench · JudgeBench Knowledge | accuracy: 50.65 to 46.49; 62.34 to 63.25 | Mixed |
| fine-tune Mistral-7B-v0.1-Instruct as Prometheus2-7b instead of using prompted judging | Mistral-7B-v0.1-Instruct · JudgeBench Overall | accuracy: 7.43 to 38.31; 6.57 to 38.31 | Helped |
Surprising findings
- Most fine-tuned LLM judges (except Skywork) score below the 50% random baseline, often due to ties, invalid outputs, and inconsistent verdicts across swapped orders.
- Judge accuracy closely tracks the same model's solving accuracy, and judges do better than solvers on math but worse on coding.
Limitations
- Responses are all generated by GPT-4o in the main split, which biases the benchmark against GPT-4o judges; the Claude-generated ablation is smaller (270 pairs).
- Small dataset size (350 pairs) with uneven category sizes, and filtering relies on GPT-4o-mini plus regex agreement.
Experimental setup
Datasets: MMLU-Pro LiveBench LiveCodeBench RewardBench MT-Bench LLMBar FairEval LLMEval-2 PRM800K HumanEvalPack
Benchmarks: JudgeBench MT-Bench LLMEval-2 FairEval LLMBar RewardBench
Metrics: accuracy
Baselines: Vanilla judge Arena-Hard Judge VertexAI Evaluation PandaLM Prometheus2 JudgeLM AutoJ Skywork judges ChatEval RewardBench reward models
Builds on: Arena-Hard AlpacaFarm LiveBench MMLU-Pro LiveCodeBench
Closest papers by meaning
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.92)
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025 · similarity 0.91)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.91)
- Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025 · similarity 0.90)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.90)
- Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.