Research map / Large language models / Alignment and preferences
UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniCBE, a uniformity-driven comparing-based evaluation framework that combines three decoupled sampling matrices (accuracy, convergence, scalability) to allocate pairwise judge budget more efficiently for LLM evaluation.
Problem
Comparing-based evaluation of LLMs with pairwise human or LLM judges is expensive (O(NM^2) comparisons), and existing budget allocation methods optimize only one objective, failing to jointly achieve accuracy, convergence, and scalability.
Approach
Derives three guidelines (uniform tuple sampling for accuracy, uniform win-rate uncertainty reduction for convergence, uniform allocation to new models for scalability), builds a sampling probability matrix for each, multiplies them into a global matrix, and samples greedily with Bradley-Terry aggregation.
Key results
- On AlpacaEval with GPT-4o judge, UniCBE saves over 17% of preference budget versus Random while reaching Pearson correlation above 0.995.
- In scenarios with continuously added models, UniCBE saves over 50% of evaluation cost compared to Random.
- Ablation shows removing the accuracy-oriented matrix causes the largest performance drop; greedy tuple sampling outperforms probabilistic sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine accuracy, convergence, and scalability allocation objectives | CBE on AlpacaEval · AlpacaEval | preference budget at Δ close to 0.01: over 17% | Helped |
| allocate budget adaptively to newly introduced models | CBE with newly added models · AlpacaEval dynamic-model scenario | evaluation budget: over 50% | Helped |
| remove the accuracy allocation matrix P^acc (ablation) | UniCBE · AlpacaEval | evaluation performance | Hurt |
| remove the convergence allocation matrix P^con (ablation) | UniCBE · AlpacaEval | Spearman correlation r_s | Hurt |
| remove the scalability allocation matrix P^sca (ablation) | UniCBE · AlpacaEval | evaluation performance | Hurt |
| replace greedy tuple sampling with probabilistic sampling (ablation) | UniCBE · AlpacaEval | evaluation performance | Hurt |
| replace Bradley-Terry aggregation with Elo rating (ablation) | UniCBE · AlpacaEval | evaluation performance | Hurt |
| replace Bradley-Terry aggregation with average win rate (ablation) | UniCBE · AlpacaEval | evaluation performance | Hurt |
| vary α within [1.5,3] (ablation) | UniCBE · AlpacaEval | performance | No clear effect |
| increase θ_acc from 1 to 2 (ablation) | UniCBE · AlpacaEval | budget savings | Mixed |
| increase sampling temperature from 0 toward +∞ (ablation) | UniCBE · AlpacaEval | evaluation performance | Hurt |
| use list-wise preference with K=3 | UniCBE · AlpacaEval list-wise preference | preference budget savings vs Random: over 30% | Helped |
Surprising findings
- AlpacaEval's fixed reference-model design performs far worse than random sampling despite being designed for scalability.
- Arena no longer outperforms Random when GPT-3.5-turbo is the judge.
Limitations
- Experiments use a random subset of 15 models and rely on API judges whose preference noise affects convergence.
- Sampling bias analysis and uniformity theory are derived under simplifying assumptions; list-wise extension shown only for K=3.
Experimental setup
Datasets: AlpacaEval MT-Bench Chatbot Arena battle data
Benchmarks: AlpacaEval MT-Bench
Metrics: mean absolute error of win rate Spearman correlation Pearson correlation budget savings vs random
Baselines: Random Arena AlpacaEval
Builds on: Bradley-Terry model Elo rating system Chatbot Arena sampling AlpacaEval
Closest papers by meaning
- Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (ICLR 2025 · similarity 0.88)
- InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.86)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.86)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.86)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.