Research map / Video / Video-language understanding
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoAutoArena, an automated arena-style benchmark where LMMs simulate users to generate open-ended video questions, judge pairwise battles, and rank 11 LMMs via ELO, plus VideoAutoBench as a streamlined companion benchmark.
Problem
Existing video understanding benchmarks rely on multiple-choice questions that poorly reflect real user queries, while human-driven arena evaluation for video is slow and expensive (about 7 samples annotated per hour).
Approach
Use LMM agents to simulate personas constrained by video content, generate user questions, judge pairwise battles between anonymous LMMs, and evolve harder questions via fault analysis of previous battles, ranking models with ELO/Bradley-Terry.
Key results
- GPT-4o achieves 87.29% agreement with human preferences as automatic judge, selected as primary judge over voting ensembles.
- Top open-source Aria (ELO 1119.99) trails GPT-4o (ELO 1505.69) by about 385.7 ELO points.
- In human blind ranking, VideoAutoArena questions ranked first in mimicking real user question style 84.2% of the time versus VideoMME and LongVideoBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply fault-driven question evolution (ablation) | VideoAutoArena evaluation of proprietary LMMs · VideoAutoArena hard prompts | ELO rating: 1512.59; 1498.04; 1316.39; 1332.29; 1177.68; 1199.56; 1144.91; 1155.77 | Mixed |
| apply fault-driven question evolution (ablation) | VideoAutoArena evaluation of open-source LMMs · VideoAutoArena hard prompts | ELO rating: 1115.21; 1126.35; 886.64; 886.10; 876.09; 874.57; 843.65; 826.79; 770.56; 762.18; 764.85; 758.66 | Mixed |
| use GPT-4o as the automatic judge | VideoAutoArena · human-annotated battles | judging accuracy: 87.29% | Helped |
| generate questions with VideoAutoArena user simulation rather than VideoMME or LongVideoBench styles | VideoAutoArena · human question-style preference comparison | first-place preference: 84.2% | Helped |
| increase question complexity using fault-driven evolution (ablation) | VideoAutoArena question generation · evolved questions | difficulty level | Helped |
| evaluate with VideoAutoArena instead of traditional multiple-choice QA | video LMM evaluation · VideoAutoArena | ELO rating: -385.7 | Mixed |
Surprising findings
- Voting with multiple LMM judges (Top2 to Top4) did not improve agreement with human preferences over GPT-4o alone.
- Open-source models' gap to proprietary models widens with longer videos and in helpfulness and relevance tracks, while Aria stays stable across lengths.
Limitations
- Judging relies on GPT-4o and human gold standard is limited to around 300 battles.
- Questions are generated by LMM user simulators rather than real users.
- Evolved questions are judged for difficulty by GPT-4o itself.
Experimental setup
Datasets: LongVideoBench
Benchmarks: VideoAutoArena VideoAutoBench
Metrics: ELO rating win rate judge accuracy vs human preference human ranking of question naturalness
Baselines: VideoMME LongVideoBench WildVision Video Arena PersonaHub Random judge Vote (Top2/Top3/Top4)
Builds on: Chatbot Arena Evol-Instruct PersonaHub MT-Bench Bradley-Terry model
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.89)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.89)
- Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs (CVPR 2025 · similarity 0.88)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.