Research map / Large language models / Language, safety and interpretability
Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prompt-to-Leaderboard (P2L), an LLM fine-tuned to map a prompt to Bradley-Terry coefficients over models, enabling prompt-specific leaderboards, routing, and strength/weakness analysis.
Problem
Aggregated LLM leaderboards (e.g., Chatbot Arena Bradley-Terry rankings) average over users and prompts, obscuring prompt-specific differences in model performance and making fine-grained evaluation data-hungry.
Approach
Train an LLM with a randomly initialized coefficient head on crowdsourced pairwise preference votes so that it outputs a prompt-conditional vector of Bradley-Terry coefficients; generalize to Prompt-to-Regression with ties and other parametric feedback models, and derive an optimal cost-constrained routing policy from the BT estimates.
Key results
- P2L-7B achieved a LiveBench overall score of 59.275 on a restricted model pool, surpassing the static oracle baseline without seeing ground-truth labels.
- The P2L router based on P2L-1.5B reached #1 on Chatbot Arena in January 2025 with a score of 1389, 25 points above Gemini-exp-1206 (1364).
- P2L aggregated leaderboards converge near optimal within 100-250 prompts, whereas marginal regression needs thousands of prompts for a stable ranking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use prompt-adaptive routing instead of a static model | P2L-1.5B router on Chatbot Arena · Chatbot Arena | Arena score: 25 points over the previous top model | Helped |
| Use P2L prompt-specific routing instead of the static oracle baseline | P2L-7B router · LiveBench | overall score: 59.275 | Helped |
| Increase the P2L router model size | P2L routers on Chatbot Arena · Chatbot Arena | Arena score | Mixed |
| Aggregate P2L predictions over prompt samples rather than use marginal regression | P2L aggregated leaderboards · Chinese and Math categories | L1 function distance: within 100-250 prompts; optimal value around 0.025 | Helped |
| Use more training data for P2L | P2L preference predictor · Chatbot Arena holdout validation set | negative log-likelihood | Helped |
| Use a larger P2L model | P2L preference predictor · Chatbot Arena holdout validation set | negative log-likelihood | Helped |
Surprising findings
- P2L-1.5B outperformed larger P2L variants in the live Chatbot Arena deployment, which the authors could not fully explain beyond statistical variation.
- Even far smaller P2L models (e.g., 1.5B) matched or exceeded top static models on LiveBench.
Limitations
- Relies on the Bradley-Terry assumption; non-transitivity breaks the optimal routing equivalence.
- Live router results were from a single ~1-week deployment on Chatbot Arena, with some unexplained variation across sizes.
- Chatbot Arena votes reflect organic user distribution, so rare categories have limited training data.
Experimental setup
Datasets: Chatbot Arena human preference votes LiveBench
Benchmarks: Chatbot Arena LiveBench
Metrics: validation loss (negative log-likelihood) mean squared error Arena score LiveBench score L1 function distance
Baselines: Marginal Bradley-Terry leaderboard Constant predictor Gemini-exp-1206 o1-2024-12-17 ChatGPT-4o-20241120 LiveBench static oracle
Builds on: Bradley-Terry model Rao-Kupper model RouteLLM RouterDC
Closest papers by meaning
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.90)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.90)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.89)
- From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline (ICML 2025 · similarity 0.88)
- Efficient multi-prompt evaluation of LLMs (NeurIPS 2024 · similarity 0.88)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.