Research map / Large language models / Alignment and preferences
Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
Jenny Y. Huang, Yunyi Shen, Dennis Wei, Tamara Broderick
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| drop worst-case 2 evaluations (0.00348%) | Chatbot Arena BT ranking · Chatbot Arena (human) | top-1 model: 2 out of 57477 flips top-1 | Hurt |
| use expert annotators and curated prompts (MT-bench vs crowdsourced arenas) | LLM arena ranking · MT-bench | fraction dropped to flip top-1: 2.74% vs 0.00348% (Chatbot Arena) | Improved |
| LLM judge instead of human judge | arena robustness · Chatbot Arena | fraction dropped to flip top-1: 0.0180% (LLM) vs 0.00348% (human) | No clear effect |
| LLM judge instead of human judge | arena robustness · MT-bench | fraction dropped to flip top-1: 1.67% (LLM) vs 2.74% (human) | Hurt |
| use bootstrap CI rankings instead of point estimate | Chatbot Arena (human) · Chatbot Arena (human) | fraction dropped to change top-1: 0.00348% -> 0.0510% | Improved |
| random 1% dropping instead of worst-case dropping (ablation) | Chatbot Arena (human) top-1 · Chatbot Arena (human) | fraction of trials top-1 unchanged: 0.77 (1.00 on most other arenas) | Improved |
Closest papers by meaning
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.91)
- Prediction-Powered Ranking of Large Language Models (NeurIPS 2024 · similarity 0.91)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.91)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.89)
- Language Ranker: A Lightweight Ranking framework for LLM Decoding (NeurIPS 2025 · similarity 0.89)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
