Research map / Large language models / Alignment and preferences
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Polyrating, a MAP-based multivariate rating system for LLM human/LLM-judge evaluation that quantifies judge biases, leverages benchmarks and LLM judges as priors to cut evaluation cost, and yields cross-task comparable ratings.
Problem
Current rating systems for LLM evaluation (Elo/Bradley-Terry style, fit per task by MLE) cannot quantify judge biases, are sample-inefficient needing large expensive preference datasets, and suffer from shift-invariance so ratings are not comparable across tasks.
Approach
Fit all preference datasets jointly with MAP estimation: a base rating per model plus task-specific modifiers and shared bias coefficients (length, position, formality, sentiment, repetitiveness, readability) with Gaussian priors; priors break shift-invariance and let LLM-judge and benchmark data act as extra games to improve sample efficiency.
Key results
- Polyrating reduces human evaluation cost for new tasks by up to 77% (hard questions) with 10000 samples, versus 58% (Chinese) and 38% (code).
- Using MixEval-Hard benchmark as prior improves sample efficiency by 41% for new models at 10000 samples; LLM-based judge (WildBench) gives 38%.
- Length bias shifts ratings by about 41 points for human judges and 48 for LLM judges; position bias adds about 38 points for LLM judges only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAP ratings that transfer information from other tasks | Polyrating · Chatbot Arena hard questions | sample efficiency: 77% | Helped |
| Leverage LLM-based preferences as an additional task | Polyrating · Chatbot Arena human evaluation | sample efficiency: 38% | Helped |
| Leverage MixEval-Hard benchmark results | Polyrating · Chatbot Arena human evaluation | sample efficiency: 41% | Helped |
| Represent a combined task with its component task features | Polyrating · Chatbot Arena Chinese code task | sample efficiency: 25% | Helped |
| Add a regularizing prior between consecutive model-version ratings | Polyrating · Chatbot Arena model versions | sample efficiency: 38% | Helped |
| Model answer length as a shared judge-bias feature | Polyrating · Chatbot Arena human evaluation | rating influence: 40.84_{\pm 2.3} | Helped |
| Model answer length as a shared judge-bias feature | Polyrating · Wildbench LLM-based evaluation | rating influence: 48.48_{\pm 1.4} | Helped |
| Model readability as a shared judge-bias feature | Polyrating · Human evaluation | rating influence: 10.75_{\pm 1.8} | Helped |
| Model readability as a shared judge-bias feature | Polyrating · LLM-based evaluation | rating influence: -3.92_{\pm 1.0} | Hurt |
| Use the Davidson draw model instead of the standard Bradley-Terry model | Polyrating · Chatbot Arena | logistic loss | No clear effect |
| Use the Rao-Kupper draw model instead of the standard Bradley-Terry model | Polyrating · Chatbot Arena | logistic loss | Hurt |
| Use the accuracy-based rating model instead of the standard Bradley-Terry model | Polyrating · Chatbot Arena | logistic loss | Hurt |
Surprising findings
- Readability increases ratings for human judges (about +11) but decreases them for LLM judges (about -4), and LLM judges are indifferent to sentiment.
- Under the univariate approach 97 of 114 models gain rating on the Chinese task, whereas Polyrating shows only 67 gain, mostly Chinese-provider models.
Limitations
- Ratings are only relative to other models in the leaderboard; no absolute performance measure without extra data such as benchmarks.
- Requires manual specification of modeling features and functions, which is time-consuming.
- Alternative draw-modeling rating systems (Rao-Kupper, Davidson) offered no clear predictive advantage, so the method keeps the BT model.
Experimental setup
Datasets: Chatbot Arena WildBench MixEval-Hard
Benchmarks: Chatbot Arena WildBench MixEval-Hard
Metrics: logistic loss sample efficiency improvement fitted bias coefficients and rating influence
Baselines: univariate Bradley-Terry (Chatbot Arena style) MLE rating Rao-Kupper model Davidson model accuracy-based model Glicko
Builds on: Bradley-Terry model Elo rating Chatbot Arena rating AlpacaEval length control
Closest papers by meaning
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.91)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.90)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.89)
- Re-evaluating Open-ended Evaluation of Large Language Models (ICLR 2025 · similarity 0.89)
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.89)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.