Research map / Adversarial robustness and security / Attacks on text and graphs
Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that voting-based LLM leaderboards like Chatbot Arena can be manipulated by de-anonymizing model responses (>95% accuracy with a BoW logistic-regression detector) and then casting targeted votes, estimating ~1000 votes to shift rankings, and evaluates mitigations such as authentication, rate limiting, and anomaly detection.
Problem
Voting-based LLM leaderboards such as Chatbot Arena rely on anonymous human votes, which can be manipulated if adversaries can identify which model produced a response and vote strategically.
Approach
A two-step reranking attack: train per-target-model classifiers on simple text features (bag-of-words, TF-IDF, length) or use identity-probing prompts to de-anonymize responses, then vote selectively for or against the target; simulate the vote/interaction cost and analyze a cost model for defenses.
Key results
- Training-based BoW detector reaches above 95% de-anonymization accuracy for most of 7 evaluated models on English prompts, and optimal prompts exceed 95% across models.
- Identity-probing prompt 'Who are you?' achieves above 90% detection accuracy for all evaluated models in the tested subset.
- Simulations estimate about 1,000 votes move a model up one position, with low-ranked models moved up 4 places needing as few as 381 votes; detector costs about $440 total.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BoW rather than TF-IDF response features (ablation) | Training-based target-model detector · English prompts | test accuracy (%): 93.7 vs 92.6; 94.7 vs 93.5; 95.8 vs 92.3; 92.8 vs 91.2; 95.7 vs 94.4; 95.7 vs 93.6; 92.0 vs 88.4 | Helped |
| Use response length features instead of BoW (ablation) | Training-based target-model detector · English prompts | test accuracy (%): 69.0 vs 93.7; 68.5 vs 94.7; 68.5 vs 95.8; 67.2 vs 92.8; 77.7 vs 95.7; 70.6 vs 95.7; 70.2 vs 92.0 | Hurt |
| Ask “Who are you?” instead of “Which model are you?” (ablation) | Identity-probing detector · claude-3-5-sonnet-20240620 | detection accuracy (%): 99.3 vs 100.0 | Hurt |
| Lower detector accuracy from 1.0 to 0.95 (ablation) | Reranking attack on llama-13b · Simulated leaderboard, target rank 79 | adversarial votes required: 1246 -> 1304 | Hurt |
| Lower detector accuracy from 0.95 to 0.9 (ablation) | Reranking attack on llama-13b · Simulated leaderboard, target rank 79 | adversarial votes required: 1304 -> 1383 | Hurt |
| Lower detector accuracy from 0.95 to 0.9 (ablation) | Reranking attack on llama-13b · Simulated leaderboard, target rank 79 | interactions required: 85000 -> 100000 | Hurt |
| Randomly upvote a model when the target is not detected instead of doing nothing (ablation) | Reranking attack on llama-13b · Simulated leaderboard, target rank 79 | interactions required: 80000 -> 75000 | No clear effect |
| Vote tie when the target is not detected instead of doing nothing (ablation) | Reranking attack on llama-13b · Simulated leaderboard, target rank 79 | interactions required: 80000 -> 80000 | No clear effect |
| Vote both responses as bad when the target is not detected instead of doing nothing (ablation) | Reranking attack on llama-13b · Simulated leaderboard, target rank 79 | interactions required: 80000 -> 80000 | No clear effect |
| Use specialized or multilingual prompts rather than ordinary high-resource-language chats (ablation) | Training-based target-model detector · Model response de-anonymization | detection accuracy | Helped |
| Increase Gaussian noise added to published Bradley-Terry ratings (ablation) | Malicious-user detection · Simulated leaderboard | detection rate | Helped |
| Increase Gaussian noise added to published Bradley-Terry ratings (ablation) | Perturbed leaderboard · Simulated leaderboard | average absolute ranking change | Hurt |
Surprising findings
- Lengths of generations alone achieve non-trivial detection accuracy, and the detector can be built for about $440 in API cost.
- Identity-probing prompts such as 'Who are you?' work well, but the leaderboard's post-processing already filters model-name mentions, limiting that attack.
Limitations
- Experiments are simulated/offline on Chatbot Arena voting logs rather than the live platform.
- A well-resourced adversary could hire many legitimate users, so mitigations raise cost but cannot fully prevent the attack.
- Detector accuracy depends on prompt choice and the set of models; results vary across models (some below 95%).
Experimental setup
Datasets: LMSYS-Chat-1M Alpaca Code MATH AdvBench Chatbot Arena voting logs (1,670,250 votes)
Benchmarks: Chatbot Arena simulated leaderboard
Metrics: detection accuracy number of votes required number of interactions required detection rate utility (average absolute rank change)
Baselines: identity-probing detector length features TF-IDF features
Builds on: Bradley-Terry model TF-IDF Chatbot Arena
Closest papers by meaning
- Improving Your Model Ranking on Chatbot Arena by Vote Rigging (ICML 2025 · similarity 0.90)
- The Leaderboard Illusion (NeurIPS 2025 · similarity 0.87)
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (NeurIPS 2024 · similarity 0.87)
- ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents (ICLR 2026 · similarity 0.87)
- Adversarial Search Engine Optimization for Large Language Models (ICLR 2025 · similarity 0.87)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.