Research map / Vision-language and multimodal / Multimodal LLMs
GenAI Arena: An Open Evaluation Platform for Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GenAI-Arena, a crowdsourced side-by-side voting platform ranking 35 open text-to-image, image editing, and text-to-video models via Elo/Bradley-Terry, and releases GenAI-Bench to test MLLM judges against human votes.
Problem
Automatic metrics (FID, CLIP, FVD) fail to capture user satisfaction with generated images and videos, and there is no trustworthy arena-style ranking for generative visual models; MLLM judges' agreement with human preferences is also unmeasured.
Approach
A Chatbot-Arena-style public voting platform covering three generative tasks, with Bradley-Terry Elo ranking, a pre-computed GenAI-Museum to avoid GPU latency, and a benchmark that scores MLLM judges by exact-match agreement with collected human votes.
Key results
- GPT-4o achieves the highest average accuracy of 49.19% judging human preferences across image generation, editing, and video generation on GenAI-Bench.
- Leaderboard top models at Oct 24, 2024: Playground V2.5 (Elo 1122) for text-to-image, MagicBrush (1108) for image editing, StableVideoDiffusion (1148) for text-to-video.
- Expert review found 93.07% of 303 valid sampled votes clearly or vaguely reasonable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o rather than Random as the MLLM judge | GenAI-Bench · GenAI-Bench | average accuracy (%): 25.47 -> 49.19 | Helped |
| Use Gemini-1.5-Pro rather than Random as the MLLM judge | GenAI-Bench · GenAI-Bench | average accuracy (%): 25.47 -> 48.94 | Helped |
| Use GPT-4o rather than Random as the MLLM judge | GenAI-Bench · Image Generation | accuracy (%): 25.36 -> 45.59 | Helped |
| Use GPT-4o rather than Random as the MLLM judge | GenAI-Bench · Image Editing | accuracy (%): 25.90 -> 53.54 | Helped |
| Use GPT-4o rather than Random as the MLLM judge | GenAI-Bench · Video Generation | accuracy (%): 25.16 -> 48.46 | Helped |
| Use Gemini-1.5-Pro rather than Random as the MLLM judge | GenAI-Bench · Image Editing | accuracy (%): 25.90 -> 55.93 | Helped |
Surprising findings
- Open-source MLLMs perform near random (25%) on GenAI-Bench, with several models scoring below random chance, e.g. Idefics1 at 2.22% average.
- Playground V2.5 wins only 0.58 of battles against PixArt-sigma despite top Elo, and Elo may be biased by imbalanced easy/hard matchups.
Limitations
- User base may be unrepresentative and votes are subjective, potentially biasing rankings.
- Battle pairing is random, causing uneven match counts across models.
- Elo estimates can be biased by imbalance between easy and hard games.
Experimental setup
Datasets: ImagenHub prompts VBench prompts GenAI-Arena user votes
Benchmarks: GenAI-Bench GenAI-Arena leaderboard
Metrics: Elo rating 95% confidence interval accuracy vs human votes expert-reviewed vote quality
Baselines: T2I-CompBench HEIM ImagenHub VBench EvalCrafter VIEScore
Builds on: Chatbot Arena ImagenHub VBench Bradley-Terry model Llama Guard
Closest papers by meaning
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.89)
- K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferences (CVPR 2025 · similarity 0.88)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.88)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.87)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.