Research map / Large language models / Language, safety and interpretability
Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Feedback Collection dataset (1K customized score rubrics, 100K GPT-4 feedback/score pairs) and fine-tunes Llama-2-Chat into Prometheus, an open-source 13B fine-grained evaluator that matches GPT-4 correlation with humans.
Problem
Proprietary LLMs like GPT-4 are the de facto evaluators for long-form text, but they are closed-source, have uncontrolled versioning, and are costly; open models lack fine-grained, customized rubric-based evaluation ability.
Approach
Train an open evaluator LM on a large dataset of GPT-4-generated feedback conditioned on thousands of customized score rubrics, reference answers, and balanced 1-5 scores, so that reference materials (rubric plus reference answer) enable GPT-4-level absolute grading.
Key results
- Prometheus 13B reaches Pearson 0.897 with human scores on 45 customized rubrics, on par with GPT-4 (0.882) and far above GPT-3.5-Turbo (0.392).
- Human annotators preferred Prometheus feedback over GPT-4 in 58.62% of pairwise comparisons and over GPT-3.5-Turbo in 79.57%.
- Prometheus 13B reaches 79.19% average accuracy on HHH Alignment, above GPT-3.5-Turbo (78.73%) and open reward models; 57.72% on MT Bench Human Judgments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama2-Chat 13B on customized-rubric feedback data | Llama2-Chat 13B · Feedback Bench (seen and unseen rubrics) | Pearson correlation with GPT-4-0613: +0.420 and +0.397 | Helped |
| Fine-tune Llama2-Chat 13B on customized-rubric feedback data | Llama2-Chat 13B · Vicuna Bench, MT Bench, and Flask Eval | Pearson correlation with GPT-4-0613: +0.255, +0.493, and +0.202 | Helped |
| Fine-tune Llama2-Chat 13B on fine-grained feedback instead of coarse-grained rubrics | Llama2-Chat 13B · HHH Alignment and MT Bench Human Judgment | accuracy: +5.43% and +5.38% | Helped |
| Add the score rubric (ablation) | Prometheus 7B · Vicuna Bench | Pearson correlation: 0.355 -> 0.457 | Helped |
| Add the score rubric (ablation) | Prometheus 7B · Feedback Bench (seen and unseen score rubrics) | Pearson correlation: 0.837 -> 0.860; 0.745 -> 0.847 | Helped |
| Add feedback distillation (ablation) | Prometheus 7B · Feedback Bench (seen and unseen score rubrics) | Pearson correlation: 0.668 -> 0.860; 0.673 -> 0.847 | Helped |
| Add feedback distillation (ablation) | Prometheus 7B · Vicuna Bench | Pearson correlation: 0.413 -> 0.457 | Helped |
| Add the reference answer (ablation) | Prometheus 7B · Feedback Bench (seen and unseen score rubrics) | Pearson correlation: 0.642 -> 0.860; 0.626 -> 0.847 | Helped |
| Add the reference answer (ablation) | Prometheus 7B · Vicuna Bench | Pearson correlation: 0.349 -> 0.457 | Helped |
| Train on coarse-grained Flask rubrics instead of customized rubrics (ablation) | Llama2-Chat 13B · Flask Eval | Pearson correlation with GPT-4-0613: 0.467 -> 0.517 | Helped |
| Train on coarse-grained rubrics (ablation) | Llama2-Chat 13B · HHH Alignment and MT Bench Human Judgment | accuracy: 73.76 -> 67.42; 52.34 -> 46.89 | Hurt |
| Use Llama-2-Chat instead of Llama-2 as the starting model (ablation) | Prometheus 7B · Feedback Bench (seen and unseen score rubrics) | Pearson correlation: 0.839 -> 0.860; 0.818 -> 0.847 | No clear effect |
Surprising findings
- Scaling Llama-2-Chat from 7B to 13B reduced GPT-4 correlation, and 70B only slightly improved it, indicating size alone does not yield evaluation ability.
- Training on coarse-grained rubrics only slightly helped or hurt performance, while fine-grained rubric training was key.
Limitations
- Prometheus remains behind GPT-4 on the GPT-4-labelled Vicuna Bench, MT Bench and FLASK Eval correlations.
- Shows a critical bias, tending to give overly critical feedback relative to humans and GPT-4.
- Human evaluation excluded math and coding instances, and the feedback dataset was generated by GPT-4 with instructions tied to rubrics.
Experimental setup
Datasets: Feedback Collection FLASK
Benchmarks: Feedback Bench Vicuna Bench MT Bench FLASK Eval MT Bench Human Judgments HHH Alignment
Metrics: Pearson Kendall-Tau Spearman accuracy pairwise win rate
Baselines: GPT-4-0613 GPT-3.5-Turbo-0613 Llama-2-Chat 70B StanfordNLP Reward Model ALMOST Reward Model Llama2-Chat 13B + Coarse
Builds on: Llama-2-Chat chain-of-thought fine-tuning FLASK
Closest papers by meaning
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.90)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.90)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.90)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.89)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.89)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.