Research map / Large language models / Language, safety and interpretability
Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation
Junjie Chen, Weihang Su, Zhumin Chu, Haitao Li, Yujia Zhou, Dingbo Yuan, Xudong Wang, Jun Zhou, Yiqun Liu, Min Zhang, Shaoping Ma, Qingyao Ai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use all three exams (consistency+pertinence+self-confidence) instead of consistency only (ablation) | Auto-PRE · Xsum/NF CATS/DailyDialog pairwise | accuracy: avg +1.45% accuracy over PRE (Auto-Exam) | Improved |
| add qualification-exam filtering of evaluators instead of using all candidates (ablation) | PRE · three datasets | accuracy: avg +2.44% accuracy over PRE (w/o Filter) | Improved |
| combine Pertinence and Self-Confidence with Consistency rather than using Pertinence alone (ablation) | Auto-PRE · DailyDialog pairwise | accuracy: 0.8065 -> 0.8161 | Improved |
| use Auto-PRE multi-evaluator ensemble instead of single GPT-4 evaluator | Auto-PRE · Xsum/NF CATS pointwise | accuracy: avg +4.53% accuracy | Improved |
| use Auto-PRE instead of GPT-4 in pairwise format | Auto-PRE · three datasets pairwise | accuracy: 0.7815 -> 0.7821 (NF CATS) | No clear effect |
| use Auto-PRE instead of GPT-4 on pairs with GPT-3.5-turbo answers | Auto-PRE · Xsum bias subset | self-preference rate: 83.64 -> 65.45 | Improved |
| replace human-annotation filter with automatic exam | PRE · three datasets pairwise | accuracy: 0.7423 -> 0.7462 (Xsum) | No clear effect |
| use Auto-PRE instead of GPT-4 as evaluator | Auto-PRE · cost analysis | cost: about 90% cost reduction, 0.54% accuracy decrease | Improved |
Closest papers by meaning
- PiCO: Peer Review in LLMs based on Consistency Optimization (ICLR 2025 · similarity 0.91)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.90)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.90)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.89)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.89)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
