Research map / Large language models / Language, safety and interpretability
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RevisEval, which has an LLM revise the response being judged and uses the revision as a response-adapted reference for LLM-as-a-Judge or classic metrics, improving evaluation accuracy and correlation.
Problem
LLM-as-a-Judge has a reliability gap with human evaluation, partly because it lacks a guiding oracle/reference; pre-existing human or direct-response references are often irrelevant to the response being evaluated (one-to-many problem), which makes reference-based evaluation noisy.
Approach
Use an LLM reviser conditioned on the instruction, rubric, and response (or, for pairwise comparison, the other response as guidance) to generate a response-adapted reference that is both higher quality and closely relevant to the response; then use it as the reference for LLM-as-a-Judge or for classic reference-based metrics.
Key results
- On NLG tasks with proprietary GPT-4 judge, RevisEval averages 34.34 Kendall tau versus 32.06 for reference-free and 32.24 for reference-based.
- On MT-Bench, AlpacaFarm, and LLMBar, GPT-4-as-a-Reviser RevisEval reaches 78.1% average accuracy versus 74.9% reference-free GPT-4 judge.
- Llama-3.1-8B judge with RevisEval reaches 70.4% average accuracy versus 59.9% reference-free and 68.8% with GPT-4 direct-response references; majority voting over classic metrics with Llama reviser reaches 61.4% vs 59.9% for Llama-as-a-Judge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use response-adapted references instead of reference-free evaluation | GPT-4-as-a-Judge · NLG tasks average | Kendall τ / Spearman ρ: 32.06/41.29 -> 34.34/43.17 | Helped |
| Use response-adapted references instead of reference-free evaluation | Llama 3.1-8B-as-a-Judge · NLG tasks average | Kendall τ / Spearman ρ: 30.79/37.90 -> 32.63/39.86 | Helped |
| Use a response-adapted reference | GPT-4-as-a-Judge · MT-Bench | accuracy: 81.2 -> 83.0 | Helped |
| Use a response-adapted reference | GPT-4-as-a-Judge · AlpacaFarm | accuracy: 70.9 -> 72.9 | Helped |
| Use a response-adapted reference | GPT-4-as-a-Judge · LLMBar | accuracy: 72.6 -> 79.0 | Helped |
| Use GPT-4 rather than Llama as the reviser (ablation) | Llama 3.1-8B-Tuned judge · MT-Bench | accuracy: 75.2 -> 79.3 | Helped |
| Use GPT-4 rather than Llama as the reviser (ablation) | Llama 3.1-8B-Tuned judge · AlpacaFarm | accuracy: 64.7 -> 67.1 | Helped |
| Use GPT-4 rather than Llama as the reviser (ablation) | Llama 3.1-8B-Tuned judge · LLMBar | accuracy: 57.8 -> 64.9 | Helped |
| Use RevisEval instead of three reference-free evaluation cycles (ablation) | Llama 3.1-8B judge · MT-Bench | accuracy: 64.1 -> 71.3 | Helped |
| Use RevisEval instead of three reference-free evaluation cycles (ablation) | Llama 3.1-8B judge · AlpacaFarm | accuracy: 54.9 -> 64.7 | Helped |
| Use RevisEval instead of three reference-free evaluation cycles (ablation) | Llama 3.1-8B judge · LLMBar | accuracy: 53.2 -> 54.9 | Helped |
| Use RevisEval instead of three reference-free evaluation cycles (ablation) | GPT-4 judge · AlpacaFarm | accuracy: 71.9 -> 72.9 | Helped |
Surprising findings
- Weak Llama-3.1-8B reviser-plus-classic-metrics majority voting outperformed the Llama-as-a-Judge baseline trained on tens of thousands of samples, suggesting generation may be more effective than further discrimination training for weak models.
- Human references hurt GPT-4 judging in story generation, decreasing Kendall correlation by about 0.06 relative to reference-free evaluation.
Limitations
- Revision for pairwise comparison uses one response as guidance, which may be imperfect; alternative revision strategies were less effective.
- Reference-based gains are small or absent for machine translation, where human references are already highly similar to responses.
- Open-source results rely on distilled GPT-4 data and a single 8B base model; the paper does not explore other base models at scale.
Experimental setup
Datasets: WebNLG-2020 WMT-22 (zh-en) SummEval OpenMEVA (ROC) MT-Bench human judgments AlpacaFarm human cross-annotations LLMBar MetricInstruct hh-rlhf JudgeBench RewardBench
Benchmarks: WebNLG-2020 WMT-22 (zh-en) SummEval OpenMEVA (ROC) MT-Bench AlpacaFarm LLMBar
Metrics: Kendall tau Spearman rho Pearson correlation accuracy positional bias change ratio
Baselines: GPT-4 reference-free judge GPT-4 reference-based judge BLEU ROUGE METEOR BERTScore BARTScore UniEval GPTScore InstructScore-7B TIGERScore-7B JudgeLM-7B
Builds on: MT-Bench LLM-as-a-Judge prompts TIGERScore MetricInstruct BERTScore Reference-guided evaluation
Closest papers by meaning
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.91)
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models (ICLR 2024 · similarity 0.89)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.89)
- CriticEval: Evaluating Large-scale Language Model as Critic (NeurIPS 2024 · similarity 0.89)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.88)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.