Research map / Vision-language and multimodal / Multimodal LLMs
VisionArena: 230k Real World User-VLM Conversations with Preference Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VisionArena, a 230K-conversation dataset of real user-VLM chats from Chatbot Arena with preference votes, plus an automatic 500-prompt benchmark (VisionArena-Bench) and a Llama-3.2-11B instruction-tuned model.
Problem
Existing VLM benchmarks are mostly static, single-turn, close-ended tasks and do not capture authentic, open-ended, multi-turn real-world user-VLM interactions or preferences.
Approach
Collects large-scale crowdsourced user-VLM conversations and pairwise preference votes from a unified LLM/VLM Chatbot Arena interface, then uses the data for style-controlled analysis, instruction tuning, and a cheap VLM-as-judge benchmark built from clustered prompts that approximates the live leaderboard.
Key results
- Fine-tuning Llama-3.2-11B on 100K VisionArena-Chat samples beats the LLaVA-Instruct version by 17 points on MMMU (26.0 vs 8.6 on MMMU... reported MMMU 23.1 vs 8.6 in table) and 46 points on WV-Bench (56.9 vs 10.4).
- VisionArena-Bench achieves 97.3% Spearman and 89.7% Kendall Tau correlation with the Chatbot Arena VLM leaderboard on 16 models, versus 80.2% and 69.2% for WV-Bench.
- Expert labels on 516 battles give Pearson 0.88 and Spearman 0.87 with the user-vote-derived BT scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-3.2-11B-Vision on VisionArena-Chat instead of Llava-Instruct-158K | Llama-3.2-11B-Vision · MMMU | Acc.: 27.9 -> 43.0 | Helped |
| Fine-tune Llama-3.2-11B-Vision on VisionArena-Chat instead of Llava-Instruct-158K | Llama-3.2-11B-Vision · WildVision-Bench | Acc.: 10.4 -> 56.9 | Helped |
| Fine-tune Llama-3.2-11B-Vision on VisionArena-Chat instead of Llava-Instruct-158K | Llama-3.2-11B-Vision · MME | Cog.: 262.1 -> 345.4 | Helped |
| Use VisionArena-Bench rather than WildVision-Bench for offline preference ranking | VLM ranking benchmark · Chatbot Arena VLM leaderboard | Spearman correlation: 80.2% -> 97.3% | Helped |
| Use VisionArena-Bench rather than WildVision-Bench for offline preference ranking | VLM ranking benchmark · Chatbot Arena VLM leaderboard | Kendall Tau correlation: 69.2% -> 89.7% | Helped |
Surprising findings
- Style factors (length, markdown) strongly shape preferences; InternVL2 ranks high on captioning largely due to style, and drops under style control.
- Top proprietary VLMs fail on some visual reasoning questions (e.g., the cat square-root meme), and spatial reasoning and planning remain weak.
Limitations
- Underrepresents geospatial, medical, and visual-assistance domains; skews toward STEM, OCR, and toy problems.
- Many languages have too few votes for stable leaderboards; preference measures user taste rather than factual accuracy.
- Dataset may still contain NSFW content or PII despite automated filtering.
Experimental setup
Datasets: VisionArena-Chat LLaVA-Instruct-158K VisionArena-Battle MMMU MME HallusionBench MMMU-Pro WildVision-Bench
Benchmarks: MME HallusionBench MMMU MMMU-Pro WV-Bench VisionArena-Bench
Metrics: accuracy Spearman correlation Kendall Tau correlation confidence agreement Bradley-Terry score Pearson correlation
Baselines: WildVision-Bench LLaVA-Instruct-158K Llama-3.2-11B-Vision-Instruct LMSYS-Chat-1M
Builds on: Chatbot Arena WildVision BERTopic Bradley-Terry model LLaVA-Instruct-158K
Closest papers by meaning
- WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences (NeurIPS 2024 · similarity 0.90)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.89)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.89)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.89)
- Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation (CVPR 2025 · similarity 0.89)
- Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations? (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.