Research map / Large language models / Agents, code and math
RocketEval: Efficient automated LLM evaluation via grading checklist
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RocketEval, which turns LLM-as-judge evaluation into instance-specific checklist grading by lightweight LLMs with normalized Yes-probabilities and optional supervised reweighting, reaching 0.965 Spearman with human Elo using Gemma-2-2B at ~2% of GPT-4o cost.
Problem
LLM-as-a-judge evaluation with powerful proprietary models is costly and hurts reproducibility and privacy, while lightweight judges show poor agreement with human preferences due to high uncertainty and positional bias.
Approach
Reframe judging as independent yes/no checklist questions generated once per instance by GPT-4o, grade each item with a lightweight LLM using conditional normalized Yes-probabilities, aggregate with an unsupervised mean or a supervised extra-trees predictor whose blend weight is set by a KL-divergence-based factor on annotation distribution.
Key results
- On WildBench, Gemma-2-2B with RocketEval achieves Spearman 0.965 vs Chatbot Arena Elo, versus 0.979 for GPT-4o with CoT and 0.923 for Llama-3-70B with direct scoring.
- Mistral-Nemo as judge with RocketEval (unsupervised and supervised) reaches Spearman 0.986 on WildBench, surpassing GPT-4o's 0.979.
- Evaluating 1000 tests on WildBench costs about $27.7 with Gemma-2-2B versus $3400 with GPT-4o, a cost reduction exceeding 50-fold for large-scale evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use instance-specific checklist grading instead of CoT scoring | Gemma-2-2B judge on MT-Bench · MT-Bench Human Judgments | agreement: 37.9% -> 57.9% | Helped |
| Use instance-specific checklist grading instead of CoT scoring | Gemma-2-2B judge on WildBench · WildBench | Spearman correlation: 0.818 -> 0.965 | Helped |
| Use supervised score prediction instead of unsupervised score prediction | RocketEval with Gemma-2-2B on WildBench · WildBench | Spearman correlation: 0.965 -> 0.965 | No clear effect |
| Remove conditional normalized score (ablation) | RocketEval with Llama-3-8B · MT-Bench Human Judgments | agreement: 63.8% -> 57.5% | Hurt |
| Judge checklist items in multi-turn format with previous judgments visible (ablation) | RocketEval with Llama-3-8B · MT-Bench Human Judgments | agreement: 63.8% -> 60.8% | Hurt |
| Set the weight factor α to the constant 1 (ablation) | RocketEval with Llama-3-8B · MT-Bench Human Judgments | agreement: 63.8% -> 59.9% | Hurt |
Surprising findings
- Chain-of-thought gives little benefit and sometimes hurts lightweight judges; e.g., Qwen2-1.5B CoT Spearman drops to 0.035 while direct scoring is 0.420.
- Llama-3.2-1B under the unsupervised CoT-free checklist method still has near-random agreement (33.6%), suggesting very small judges remain unreliable even with checklists.
Limitations
- Checklist creation depends on GPT-4o, so the pipeline still incurs proprietary-model cost and dependency for the one-time generation step.
- Supervised predictor requires human or GPT-4o annotations on the benchmark, limiting pure zero-shot use; very small judges (e.g., Llama-3.2-1B) still perform poorly.
- Correlation evaluation uses a small set of about 12 to 50 test models and the Elo ground truth has overlapping confidence intervals in some cases, so rankings are sensitive to model selection.
Experimental setup
Datasets: MT-Bench MT-Bench Human Judgments WildBench Chatbot Arena Elo ratings AlpacaEval Arena-Hard
Benchmarks: MT-Bench Human Judgments WildBench MT-Bench AlpacaEval Arena-Hard
Metrics: agreement ratio Spearman correlation Kendall's tau cost (USD)
Baselines: GPT-4o GPT-4 Prometheus-7B-v2.0 Claude-3.5-Sonnet Arena-Hard-Auto
Builds on: WildBench MT-Bench Prometheus
Closest papers by meaning
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.92)
- Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks (ICLR 2025 · similarity 0.92)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.91)
- Tuning LLM Judge Design Decisions for 1/1000 of the Cost (ICML 2025 · similarity 0.91)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.91)
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.