Research map / Large language models / Language, safety and interpretability

Improving Your Model Ranking on Chatbot Arena by Vote Rigging

ICML 2025 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that Chatbot Arena's crowdsourced votes can be rigged to raise or lower a target model's rank, and proposes omnipresent rigging strategies (Omni-BT, Omni-On) that exploit Elo/Bradley-Terry coupling so every new vote can help, using a multi-class classifier to de-anonymize models.

Problem

Chatbot Arena ranks LLMs from crowdsourced pairwise votes, which are assumed reliable; the paper asks whether an adversary can manipulate a target model's leaderboard rank by casting strategically chosen votes, and how efficiently.

Approach

Target-only rigging votes only when the target model appears in a battle, which is inefficient (~1% of battles). Omnipresent rigging exploits the fact that under Bradley-Terry fitting, any vote between two other models shifts the target's score, so every new vote is manipulated: a multi-class classifier de-anonymizes both models and the vote is chosen to maximize a rigging objective (BT-score gap

Key results

• Omni-BT and Omni-On achieve roughly double the ranking promotion of target-only rigging; with 20,000 idealized rigged votes, omnipresent strategies reach about 10-rank promotion versus about 4 for target-only.
• Omni-BT rigging on the real-world-like Chatbot Arena simulation gives an average 5-rank promotion and Omni-On 3-rank, over 50% more than the best target-only T-Abstain.
• Vote filtering with win-rate threshold tau=0.7 still leaves omnipresent strategies at over 6-rank improvement.

Measured effects reported in the paper

Surprising findings

• Target-only rigging with T-Tie or T-Random yields no manipulation effect when half of battles have anonymous identities.
• Updating local BT scores during Omni-On manipulation performs worse than keeping the initial leaderboard scores fixed.

Limitations

• Experiments are simulations on a historical vote dataset rather than the live Chatbot Arena platform.
• The multi-class classifier covers a fixed set of known models; unrecognized new models reduce accuracy, though rigging remains partially effective.
• Defenses reduce but do not eliminate the rigging effect, and adaptive adversaries can evade some of them.

Experimental setup

Datasets: Chatbot Arena historical votes (clean_battle_20240814_public, ~1.7M votes, 129 models) HC3 Quora

Benchmarks: Chatbot Arena historical votes simulation (Chatbot Arena Notebook)

Metrics: ranking improvement (rank positions) top-1 and top-5 classifier accuracy

Baselines: T-Tie T-Abstain T-Random T-Normal w/o Rigging

Builds on: Bradley-Terry model Elo rating system RoBERTa

Closest papers by meaning

• Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards (ICML 2025 · similarity 0.90)
• A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.88)
• The Leaderboard Illusion (NeurIPS 2025 · similarity 0.88)
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.88)
• Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations (ICML 2025 · similarity 0.85)
• WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.85)