Research map / Image generation / Text-to-image diffusion
K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes K-Sort Arena, a human-preference arena for visual generative models that uses K-wise free-for-all votes, Bayesian probabilistic rating updates, and UCB-based matchmaking to converge faster than Elo.
Problem
Pairwise Arena evaluation (e.g., Chatbot Arena with Elo) needs very many votes to converge and is sensitive to preference noise, making it slow to rank text-to-image and text-to-video models as new ones appear.
Approach
Replace pairwise battles with K-wise (K=4 default) free-for-all votes, model each model's capability as a Gaussian updated by Bayesian posterior inference with an added observation-noise term, and choose opponents via a UCB exploration-exploitation rule with a balance-guided pivot choice.
Key results
- K-Sort Arena reaches convergence in 716 comparisons versus 11692 for the Elo system, a 16.3x reduction in simulation.
- K=4 K-wise comparisons converge roughly twice as fast as pairwise (K=2) under UCB matchmaking.
- Over 1,000 crowdsourced votes from visual generation experts were collected to build text-to-image and text-to-video leaderboards.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use K-Sort Arena with K=4, probabilistic modeling, and UCB matchmaking instead of the Elo system | Simulated ranking from scratch · Simulated ranking from scratch | comparisons to convergence: 11692 -> 716 (↓16.3×) | Helped |
| Use probabilistic modeling instead of numerical modeling (ablation) | K=2 ranking without preference noise · Simulated ranking from scratch, no preference noise | MSE: about 1500 comparisons | Helped |
| Use probabilistic modeling instead of numerical modeling (ablation) | K=2 ranking with 5% preference noise · Simulated ranking from scratch, 5% preference noise | MSE: approximately 2000 comparisons | Helped |
| Increase comparison size from K=2 to K=4 (ablation) | K-Sort Arena with UCB matchmaking · Simulated ranking from scratch | ranking convergence speed: approximately twice as fast | Helped |
| Increase comparison size from K=4 to K=6 (ablation) | K-Sort Arena with UCB matchmaking · Simulated ranking from scratch | ranking convergence speed | Mixed |
| Use UCB matchmaking instead of random matchmaking (ablation) | K=4 ranking · Simulated ranking from scratch | MSE: 3,000 comparisons | Helped |
| Use UCB matchmaking instead of skill-based matchmaking (ablation) | K=4 ranking · Simulated ranking from scratch | MSE | Helped |
| Use balance-guided pivot selection instead of a random pivot (ablation) | K-Sort Arena with UCB opponent matching · Adding a new model to a ranking of 50 models | comparisons to determine the new model's rank: roughly 30 comparisons | Helped |
Surprising findings
- Numerical Elo modeling fails to converge even after 3000 comparisons, while probabilistic modeling converges after about 1500.
Limitations
- Headline efficiency numbers come from simulated votes with preset labels rather than real user data.
- Voters are limited to professors and graduate students in visual generation, a narrow pool.
- Sora was evaluated only on official samples due to lack of API access.
Experimental setup
Datasets: MS COCO WebVid
Benchmarks: K-Sort Arena crowdsourced leaderboard (text-to-image and text-to-video)
Metrics: number of comparisons to convergence mean squared error of ranks conservative score
Baselines: Elo system (Chatbot Arena) TrueSkill skill-based matchmaking random matchmaking
Builds on: Elo TrueSkill Glicko Chatbot Arena UCB
Closest papers by meaning
- GenAI Arena: An Open Evaluation Platform for Generative Models (NeurIPS 2024 · similarity 0.88)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.86)
- SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset (NeurIPS 2024 · similarity 0.86)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.85)
- Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.85)
- DrPO: Drifting Preference Optimization for One-Step Generative Models (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.