Research map / Vision-language and multimodal / Multimodal LLMs
WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WildVision-Arena, an online human-preference platform for VLMs, and WildVision-Bench, a 500-sample in-the-wild benchmark judged by GPT-4o against Claude-3-Sonnet that correlates with Elo rankings (Spearman 0.94).
Problem
Existing VLM benchmarks do not track human preferences over many models in real-world multimodal chats, and fixed benchmarks fail to capture the gap between proprietary and open models seen in user experience.
Approach
Crowdsource pairwise human votes on multi-round image chats across 20+ VLMs to compute Bradley-Terry/Elo rankings, then curate a small benchmark from the same in-the-wild data and score models with an AlpacaEval-style GPT-4o judge versus Claude-3-Sonnet so that benchmark rankings track arena rankings.
Key results
- WV-Bench model scores achieve a Spearman correlation of 0.94 with WildVision-Arena Elo, outperforming MMVet, MMMU, and MMStar.
- GPT-4o ranks first on the arena with Elo 1235 and a model score of 89.41 on WV-Bench, versus 80.01 for GPT-4V.
- Expert-user agreement on 100 samples averages 72.5% with Cohen's Kappa 0.59.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o to judge model answers against Claude-3-Sonnet on WV-Bench | WV-Bench model ranking · WV-Arena Elo | Spearman correlation: 0.94 | Helped |
Surprising findings
- GPT-4V has relatively low agreement with humans on tie votes, and predicting when both models are bad is challenging for the GPT-4V judge.
- Open-source LLaVA-NEXT-34B outranks Gemini-1.5-Flash and Claude-3-Sonnet on the arena leaderboard.
Limitations
- Hidden set is only 50 expert-annotated samples and is not publicly released, limiting reproducibility.
- Judge models show low agreement with humans on tie and both-bad votes.
- Failure analysis relies on sampled cases rather than systematic annotation.
Experimental setup
Datasets: WildVision-Arena user chats and votes
Benchmarks: WildVision-Bench MMVet MMMU MMStar MMBench MathVista HallusionBench
Metrics: Elo rating Spearman correlation model score with 95% CI win rate Cohen's kappa
Baselines: MMVet MMMU MMStar MMBench MathVista HallusionBench
Builds on: Chatbot Arena AlpacaEval Bradley-Terry model
Closest papers by meaning
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
- VisionArena: 230k Real World User-VLM Conversations with Preference Labels (CVPR 2025 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.90)
- SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.90)
- Can We Talk Models Into Seeing the World Differently? (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.