Research map / Multi-agent systems and games / LLM agents
Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Ego2Web, a 500-task benchmark pairing egocentric videos with live web tasks, plus Ego2WebJudge, an MLLM-as-judge that grounds web-agent success in video evidence.
Problem
Existing web-agent benchmarks ignore the user's physical surroundings, so they cannot evaluate agents that must perceive egocentric video and then act on the web.
Approach
Synthesize video-grounded web tasks from Qwen3-VL captions via an LLM planner, with human verification, and evaluate agents on live websites with an LLM judge that additionally uses annotated egocentric keyframes/clips and keypoints (extending WebJudge).
Key results
- BU-Gemini-3-Flash achieves the best human-evaluated success rate of 58.6% on Ego2Web, versus 26.4% to 44.4% for other agents except it is top; Claude 3.7 gets 26.4%.
- Ego2WebJudge reaches 80.8% average agreement with human judgment using Gemini-2.5 Pro and 84.0% with GPT-4o, above WebJudge (76.1%, 78.4%) and WebVoyager (70.7%, 74.7%).
- Removing raw video drops BU-Gemini-3-Flash success from 48.2% to 4.4% without visual input and 23.6% with detailed captions only.
Surprising findings
- Detailed captions recover only about half of the gain from raw video input (23.6% vs 48.2% SR), indicating text abstractions lose critical visual detail.
- Claude-based agents show high agreement across evaluators mainly because they rarely succeed, inflating agreement rates.
Limitations
- Benchmark is small at 500 video-instruction pairs across 18 websites.
- Agents without native video input (Claude, GPT-5.4) receive Gemini-generated captions, confounding comparisons.
- Judge agreement results depend on the chosen MLLM backbone and the judge may still err on fine-grained visual details.
Experimental setup
Datasets: Ego4D EgoSchema clips
Benchmarks: Ego2Web
Metrics: success rate agreement rate with human evaluation
Baselines: SeeAct Browser Use with GPT-4.1 Browser Use with Gemini-3-Flash Claude Sonnet 3.7 Computer Use Claude Sonnet 4.5 Computer Use GPT-5.4 WebVoyager WebJudge
Builds on: WebJudge WebVoyager Ego4D EgoSchema
Closest papers by meaning
- EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents (NeurIPS 2026 · similarity 0.91)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.89)
- VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks (ICLR 2025 · similarity 0.88)
- Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.87)
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.86)
- AgenticVBench: Can AI Agents Complete Real-World Video Production Tasks? (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.