Research map / Large language models / Alignment and preferences
Investigating Non-Transitivity in LLM-as-a-Judge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LLM judges (GPT-4-Turbo, GPT-3.5-Turbo) produce non-transitive pairwise preferences in AlpacaEval, makes baseline-fixed rankings unstable, and proposes round-robin Bradley-Terry tournaments plus the Swiss-wise iterative matchmaking (SWIM) method to get more robust rankings.
Problem
Baseline-fixed pairwise LLM-as-a-judge evaluation (e.g., AlpacaEval) assumes transitive preferences, which is unverified; non-transitivity can make model rankings sensitive to the choice of baseline.
Approach
Introduce Soft Non-Transitivity Deviation (SNTD) to measure continuous non-transitivity in judge preferences, and replace fixed-baseline ranking with round-robin tournaments scored by Bradley-Terry with soft labels, plus a logarithmic-cost Swiss-style matchmaking (SWIM) variant.
Key results
- Round-robin Bradley-Terry ranking raises Spearman correlation with Chatbot Arena from 95.0% to 96.4% and Kendall from 82.1% to 86.3% under length control.
- Only 20% of models keep consistent rank positions across all 20 baseline-fixed ranking lists, showing high sensitivity to baseline choice.
- Non-transitivity is highest when compared models have similar performance; GPT-3.5-Turbo shows over 96% of instructions affected by position bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use round-robin comparisons instead of fixed-baseline comparisons | AlpacaEval ranking · Chatbot Arena, with length control | Spearman correlation: 95.0% -> 96.4% | Helped |
| Use round-robin comparisons instead of fixed-baseline comparisons | AlpacaEval ranking · Chatbot Arena, with length control | Kendall correlation: 82.1% -> 86.3% | Helped |
| Use round-robin comparisons instead of fixed-baseline comparisons | AlpacaEval ranking · Chatbot Arena, without length control | Spearman correlation: 81.4% -> 85.4% | Helped |
| Use round-robin comparisons instead of fixed-baseline comparisons | AlpacaEval ranking · Chatbot Arena, without length control | Kendall correlation: 63.2% -> 68.4% | Helped |
| Use length control | AlpacaEval 2.0 · Chatbot Arena | Spearman correlation: 81.4% -> 95.0% | Helped |
| Use length control | AlpacaEval 2.0 · Chatbot Arena | Kendall correlation: 63.2% -> 82.1% | Helped |
| Add systematic position switching instead of random position assignment | GPT-4-Turbo judge · AlpacaEval, ambiguous instructions | Non-transitive case rate: 17% to 44% | Helped |
| Add systematic position switching instead of random position assignment | GPT-3.5-Turbo judge · AlpacaEval, ambiguous instructions | Non-transitive case rate | Hurt |
| Use a structured evaluation checklist | LLM judge · AlpacaEval, MM | Non-transitive case rate | Helped |
| Add Chain-of-Thought reasoning | LLM judge · AlpacaEval, MM | Non-transitive preference rate | Hurt |
| Allow the judge to declare ties | LLM judge · AlpacaEval, MM | Non-transitive preference rate | Hurt |
| Replace position switching and two API calls per order with random assignment and a single API call (ablation) | Model-level pairwise comparisons · AlpacaEval, 20-model ranking | Hard non-transitivity | Hurt |
Surprising findings
- Position switching reduces non-transitivity for GPT-4-Turbo but random assignment can reduce it less or even increase it for ambiguous instructions; GPT-3.5-Turbo shows the opposite effect.
- Chain-of-thought prompting mitigates position bias but increases non-transitive preferences for the judge.
Limitations
- Evaluated only on GPT-family judges and AlpacaEval/Arena-Hard instructions, limiting generality.
- Bradley-Terry assumes global transitive model scores, which may not capture multi-dimensional capability.
- Human rankings from Chatbot Arena may themselves contain non-transitivity.
Experimental setup
Datasets: AlpacaEval dataset Arena-Hard-Auto prompts Chatbot Arena human rankings (reference)
Benchmarks: AlpacaEval Arena-Hard-Auto Chatbot Arena
Metrics: PNT SNTD Spearman correlation Kendall correlation rank consistency
Baselines: AlpacaEval 2.0 (baseline-fixed win rate) Length-Controlled AlpacaEval Chatbot Arena rankings (reference)
Builds on: AlpacaEval Bradley-Terry model Length-Controlled AlpacaEval
Closest papers by meaning
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.89)
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.89)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.88)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.88)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.88)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.