Research map / Video / Video-language understanding
VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoWebArena, a 2,021-task web-agent benchmark built on 74 manually created video tutorials, testing long-context video understanding via factual and skill retention tasks.
Problem
Existing agent benchmarks focus on text and static images and neglect long-context video understanding, so agents' ability to use video tutorials and retrieve information from videos to complete web tasks is unmeasured.
Approach
Builds a benchmark where every task requires watching a video: skill retention tasks reuse WebArena/VisualWebArena intents paired with tutorial videos, and factual retention tasks add incidental-detail questions with a separate intermediate video-QA check that separates perception failures from action failures.
Key results
- Best agent (GPT-4o summary agent) reaches 13.3% final success on factual retention tasks vs 73.9% for humans.
- GPT-4o 30/100-frame agent reaches 45.8% intermediate VQA success vs 79.3% for humans.
- On skill retention, GPT-4o with tutorial summary drops from 14.9% to 13.8% on WebArena and from 19.8% to 11.6% on VisualWebArena versus no tutorial.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add tutorials in-context to GPT-4o | GPT-4o on WebArena skill retention · WebArena | final success rate: 14.9% -> 13.8% | Hurt |
| add tutorials in-context to GPT-4o | GPT-4o on VisualWebArena skill retention · VisualWebArena | final success rate: 19.8% -> 11.6% | Hurt |
| use 30 video frames instead of a task-conditioned summary | GPT-4o on VideoWebArena factual retention · VideoWebArena factual retention | final success rate: 13.3% -> 11.0% | Hurt |
| use 30 video frames instead of a task-conditioned summary | GPT-4o on VideoWebArena factual retention · VideoWebArena factual retention | intermediate success rate: 36.8% -> 45.8% | Helped |
| increase sampled frames from 30 to 60 (ablation) | GPT-4o Frame Agent on VideoWebArena factual retention · VideoWebArena factual retention | final success rate: 11.0% -> 6.0% | Hurt |
| increase sampled frames from 30 to 60 (ablation) | GPT-4o Frame Agent on VideoWebArena factual retention · VideoWebArena factual retention | intermediate success rate: 45.8% -> 43.5% | Hurt |
| increase sampled frames from 60 to 100 (ablation) | GPT-4o Frame Agent on VideoWebArena factual retention · VideoWebArena factual retention | final success rate: 6.0% -> 9.5% | Helped |
| increase sampled frames from 60 to 100 (ablation) | GPT-4o Frame Agent on VideoWebArena factual retention · VideoWebArena factual retention | intermediate success rate: 43.5% -> 45.8% | Helped |
| add tutorials to human task completion | Humans on WebArena skill retention · WebArena | final success rate: 82.6% -> 93.1% | Helped |
| add tutorials to human task completion | Humans on VisualWebArena skill retention · VisualWebArena | final success rate: 72.7% -> 88.6% | Helped |
Surprising findings
- Adding video tutorials in context hurts agent performance on skill retention tasks compared with no tutorial.
- The summary agent, which never sees visual video content, achieves the best final task success on factual retention.
Limitations
- Gemini agent evaluated on full set but GPT-4o only on a smaller subset due to compute constraints.
- Frame-based baselines miss visual information between sampled frames, a flaw in the baseline setup.
- All videos are in English and created by a small group of graduate students, introducing possible linguistic and cultural bias.
Experimental setup
Datasets: WebArena tasks VisualWebArena tasks
Benchmarks: VideoWebArena WebArena VisualWebArena
Metrics: final task success rate intermediate VQA success rate average steps
Baselines: GPT-4o Summary Agent GPT-4o Frame Agent Gemini 1.5 Pro Video Agent Phi-3.5V Human performance
Builds on: WebArena VisualWebArena Set-of-Marks prompting
Closest papers by meaning
- AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.89)
- AgenticVBench: Can AI Agents Complete Real-World Video Production Tasks? (NeurIPS 2026 · similarity 0.88)
- Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.88)
- Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos (CVPR 2026 · similarity 0.88)
- ReFAct: Empowering Multimodal Web Agents with Visual and Context Focusing (CVPR 2026 · similarity 0.88)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.