Research map / Large language models / Language, safety and interpretability
Fewer Battles, More Gain: An Information-Efficient Framework for Arena-based LLM Evaluation
Zirui Liu, Xianquan Wang, Yan Zhuang, Jiatong Li, Qi Liu, Shuanghong Shen, Mingyue Cheng, Shijin Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select pairs by D-optimality instead of random | arena pair selection · Chatbot+PPE total | pairwise index: .8409 -> .8531 | Improved |
| select pairs by A-optimality instead of random | arena pair selection · Chatbot+PPE total | pairwise index: .8409 -> .8456 | No clear effect |
| select pairs by Nearest ability instead of random | arena pair selection · Chatbot+PPE total | pairwise index: .8409 -> .8410 | No clear effect |
| select pairs by Ada-Pair instead of random | arena pair selection · Chatbot+PPE total | pairwise index: .8409 -> .8437 | No clear effect |
| use time-improved top-10 selection for D-optimality (ablation) | D-optim · Chatbot+PPE total | pairwise index: .8531 -> .8581 | Improved |
| use time-improved top-10 selection for A-optimality (ablation) | A-optim · Chatbot+PPE total | pairwise index: .8456 -> .8434 | No clear effect |
| use time-improved top-10 selection for Ada-Pair (ablation) | Ada-Pair · Chatbot+PPE total | pairwise index: .8437 -> .8421 | No clear effect |
| use time-improved top-10 selection for Nearest (ablation) | Nearest · Chatbot+PPE total | pairwise index: .8410 -> .8459 | No clear effect |
| use m-ELO (MLE) instead of ELO ratings | D-optim · Chatbot+PPE | pairwise index: .8486 -> .8577 | Improved |
| select pairs by D-optimality instead of random on larger pool | arena pair selection · Arena-100k total | pairwise index: .8249 -> .8346 | Improved |
| select pairs by D-optimality instead of Ada-Pair | arena pair selection · Arena-100k total | pairwise index: .8312 -> .8346 | Improved |
Closest papers by meaning
- am-ELO: A Stable Framework for Arena-based LLM Evaluation (ICML 2025 · similarity 0.91)
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.90)
- Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (ICLR 2025 · similarity 0.89)
- Elo Uncovered: Robustness and Best Practices in Language Model Evaluation (NeurIPS 2024 · similarity 0.89)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.89)
- Re-evaluating Open-ended Evaluation of Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
