Research map / Large language models / Alignment and preferences
Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Control Variates Evaluation, which uses synthetic LLM/reward-model feedback as a control variate to cut the number of human annotations needed for unbiased head-to-head win-rate estimation, with optional finetuning of the synthetic evaluator.
Problem
Head-to-head LLM win-rate evaluation with human feedback is costly and can degrade user experience, while synthetic (LLM-as-a-judge or reward model) feedback is cheap but biased.
Approach
Apply the classical control variates technique so that a biased synthetic evaluator reduces variance of the human-based win-rate estimate while keeping it unbiased; the variance reduction equals the squared correlation between human and synthetic preferences, giving a predictable human annotation saving ratio. Optionally finetune the reward model on existing human annotations to raise this correla
Key results
- Finetuned Skywork-8B achieves an averaged human annotation saving of 24.8% on Chatbot Arena and 12.6% on MT Bench, versus 8.3% and 7.5% for the pretrained model.
- Off-the-shelf ArmoRM-8B and GPT-4 both reach 12.2% saving on Chatbot Arena; GPT-4 reaches 11.9% on MT Bench.
- Finetuning GRM-2B raises saving from 10.6% to 17.1% on Chatbot Arena; in LLM-as-a-judge setting GRM-2B saves 22.8% of GPT-4 annotations on Chatbot Arena.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune GRM-2B on existing human annotations (ablation) | Control Variates Evaluation with GRM-2B · Chatbot Arena | human annotation saving ratio: 10.6% -> 17.1% | Helped |
| finetune GRM-2B on existing human annotations (ablation) | Control Variates Evaluation with GRM-2B · MT Bench | human annotation saving ratio: 5.7% -> 10.9% | Helped |
| finetune Skywork-8B on existing human annotations (ablation) | Control Variates Evaluation with Skywork-8B · Chatbot Arena | human annotation saving ratio: 8.3% -> 24.8% | Helped |
| finetune Skywork-8B on existing human annotations (ablation) | Control Variates Evaluation with Skywork-8B · MT Bench | human annotation saving ratio: 7.5% -> 12.6% | Helped |
| use Control Variates Evaluation instead of Synthetic Evaluation alone | LLM win-rate evaluation · Chatbot Arena | mean square error | Helped |
| use a small reward model as the control variate with GPT-4 as the strong evaluator | LLM-as-a-judge evaluation · Chatbot Arena | GPT-4 annotation sample saving: 22.8% | Helped |
Surprising findings
- Synthetic evaluators that are themselves highly biased (e.g., GPT-4 alone) still reduce human annotation needs when used as control variates.
- Estimating the control variates coefficient alpha from the same human samples introduces negligible bias in practice.
Limitations
- Only studies canonical head-to-head win-rate between two LLMs; multi-model ranking and fine-grained metrics not explored.
- Finetuning adds computation cost and requires existing human annotations on a benchmark.
- Savings are modest (roughly 8-25%) relative to full human evaluation.
Experimental setup
Datasets: Chatbot Arena MT Bench
Benchmarks: Chatbot Arena MT Bench
Metrics: human annotation saving ratio mean square error bias
Baselines: Human Evaluation Synthetic Evaluation GPT-4 as judge Human (shifted)
Builds on: control variates prediction-powered inference Bradley-Terry model
Closest papers by meaning
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.92)
- Human Feedback is not Gold Standard (ICLR 2024 · similarity 0.91)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.90)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.90)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.90)
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.