Research map / Large language models / Alignment and preferences

A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation

NeurIPS 2026 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an influence-based perturbation framework for Bradley-Terry leaderboards that audits robustness (top-k, CI-aware top-k, Kendall's tau, CI width) under Drop/Add/Flip match perturbations and player removal, and uses the same scores for targeted manipulation.

Problem

Leaderboards such as Chatbot Arena aggregate pairwise preferences into Bradley-Terry rankings, but how small structured changes to the comparison data affect top-k membership, global ranking, and uncertainty is poorly understood.

Approach

Treat match drops, additions, flips and player removals as weight perturbations to the weighted BT M-estimator and propagate influence scores (with a one-step Newton refinement) to several leaderboard objectives, unifying robustness auditing and targeted manipulation in one formalism.

Key results

• On Arena 55k, 3 Flip or 5 Drop actions (0.01% of matches) change the Top-1 model under a 5% budget.
• Influence-guided Add reduces target-model CI width more than Arena Active and Random; averaged over datasets, -0.37% vs -0.32% vs -0.02% after 12 actions.
• Influence-guided promotion/demotion uses about 74 actions on average versus about 92 for omni-rigging (~19% reduction).

Surprising findings

• Random perturbations barely change Kendall's tau or trace uncertainty even at budget 30, so influence-guided selection is essential.
• Removing the single most influential player can move up to 28 players and change up to 6 top-10 memberships.

Limitations

• Influence estimates are local linear approximations that may be less accurate under large budgets or nonlinear ranking changes.
• Uncertainty objectives are BT-based proxies that do not capture annotation noise, judge bias, or prompt-level dependence.
• Dual-use risk: the same techniques can be used to manipulate rankings.

Experimental setup

Datasets: Arena-55K (arena-human-preference-55k) Chatbot Arena LLM Judges MT-Bench human judgments Vision Arena WebDev Arena NBA Elo top-50 ATP tennis top-10

Benchmarks: Arena-55K Chatbot Arena LLM Judges MT-Bench human judgments Vision Arena WebDev Arena NBA Elo top-50 ATP tennis top-10

Metrics: minimum actions to change top-1/top-k Kendall's tau confidence interval width change normalized robustness score

Baselines: omni-rigging (Omni-On) Arena Active sampling Random selection

Builds on: Bradley-Terry model influence functions omni-rigging one-step Newton case-deletion

Closest papers by meaning

• Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations (ICML 2025 · similarity 0.85)
• The Leaderboard Illusion (NeurIPS 2025 · similarity 0.85)
• Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards (ICML 2025 · similarity 0.85)
• Improving Your Model Ranking on Chatbot Arena by Vote Rigging (ICML 2025 · similarity 0.84)
• A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.84)
• From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.83)