Research map / Large language models / Reasoning and chain of thought
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JETTS, a benchmark that evaluates LLM-as-judge models as evaluators in three test-time scaling tasks (reranking, step-level beam search, critique-based refinement) across math, code, and instruction following.
Problem
LLM-judges are popular for automatic evaluation, but their effectiveness as evaluators guiding test-time scaling (versus scalar reward models) is largely unknown.
Approach
Instead of fixed pairwise test sets, the benchmark measures judges by the improvement they bring to a generator in simulated test-time scaling procedures (best-of-N reranking, beam search, and critique-driven refinement), using normalized helpfulness and effective improvement ratio metrics.
Key results
- Judges are competitive with outcome reward models in pairwise reranking but consistently worse than the 7B process reward model Qwen2.5-Math-PRM in beam search.
- No judge yields effective improvement ratio above 1.0 in critique-based refinement; instruction following shows up to 10% degradation.
- Judges show highest helpfulness on instruction following, mostly positive on math, and mostly negative on code; larger judge-to-generator size ratio helps math but not code.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pairwise round-robin rather than single-instance rating for reranking | LLM judges · GSM8K, MATH, CHAMP, HumanEval+, MBPP+, BigCodeBench, AlpacaEval, IFEval | normalized helpfulness | Mixed |
| Use pairwise reranking | LLM judges · GSM8K and MATH | normalized helpfulness | Helped |
| Use pairwise reranking | LLM judges · AlpacaEval | normalized helpfulness | Helped |
| Increase judge-to-generator parameter ratio | LLM judges in reranking · Math reasoning | normalized helpfulness | Helped |
| Increase judge-to-generator parameter ratio | LLM judges in reranking · Code generation | normalized helpfulness | No clear effect |
| Use larger judges relative to the generator | LLM judges in beam search · Math reasoning | normalized helpfulness | Helped |
| Use LLM judges instead of Qwen2.5-Math-PRM-7B for beam search | Beam search evaluators · Math reasoning | normalized helpfulness | Hurt |
| Evaluate completed lookahead responses rather than partial responses | Beam search evaluators · Math reasoning and code generation | normalized helpfulness | Mixed |
| Provide judge-generated critiques and use them for iterative response refinement | SFR-Judge, Prometheus, and Themis · Math reasoning, code generation, and instruction following | effective improvement ratio | Hurt |
| Use larger judges relative to the generator | LLM judges in reranking · Instruction following | normalized helpfulness | Helped |
Surprising findings
- Refinement with judge critiques fails to beat both reranking and greedy decoding on any dataset, with no judge improving over baseline.
- Single-instance rating judges are overly lenient, rating many responses top score, making reranking outcomes swing from very poor to near oracle depending on tie-breaking.
Limitations
- Beam search and refinement require substantially more compute, limiting the number of judges and settings evaluated.
- Judges were not trained for partial-response judging, so step-level evaluation is out-of-distribution for them.
- Lookahead beam search results are inconclusive and depend on judge strength.
Experimental setup
Datasets: GSM8k MATH CHAMP HumanEval+ MBPP+ BigCodeBench AlpacaEval IFEval
Benchmarks: GSM8k MATH CHAMP HumanEval+ MBPP+ BigCodeBench AlpacaEval IFEval
Metrics: normalized helpfulness effective improvement ratio accuracy pass@1 win rate
Baselines: Skywork-Reward-v0.2 OffsetBias-RM Qwen2.5-Math-PRM Llama-3.1-8B-Instruct with judge prompt majority vote greedy decoding
Builds on: RewardBench Self-consistency AlpacaEval Math-Verify BigCodeBench
Closest papers by meaning
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.92)
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge (ICML 2025 · similarity 0.90)
- Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions (AAAI 2025 · similarity 0.89)
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (NeurIPS 2025 · similarity 0.89)
- Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.89)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.