Research map / Multi-agent systems and games / LLM agents
EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoBench, an interactive egocentric-video benchmark (1,590 tasks, five scenarios) for tool-using multimodal agents, with a multi-agent simulated user and a joint process/result validation metric; eight video-MLLMs score at most 34.95% joint success.
Problem
Existing benchmarks do not jointly evaluate multimodal perception, multi-hop tool-using reasoning, and dynamic user interaction, and lack objective evaluation of interactive, multimodal agent behavior.
Approach
Tasks are grounded in egocentric videos with a visual-data information gap so that success requires both perception and tool calls; a multi-agent (Actor-Evaluator-Summarizer) simulated user provides task-aligned feedback; and a deterministic joint metric checks tool-call coverage plus final database-state equivalence.
Key results
- Best model Gemini-3.1-Pro reaches 34.95% average Joint Success Rate across the three interaction modes (36.21% Easy, 27.03% Hard, 41.60% Static).
- Multi-agent simulated user raises All Success from 0.8935 to 0.9772 with Qwen3.5-397B-A17B as backbone.
- Best open-source model Qwen3.5-397B-A17B reaches 19.14% average Joint Success Rate.
Surprising findings
- Static mode did not improve accuracy over dynamic easy mode, and some models performed worse in static mode, suggesting multi-turn interaction compensates for planning limits.
- Failures are dominated by multimodal perceptual misinterpretations and logical fallacies rather than output-format errors.
Limitations
- Performance ceiling is low for all models, and the benchmark covers only five scenarios and 1,590 tasks.
- Simulated user relies on a single LLM backbone (Qwen3.5-397B-A17B) for driving interactions, with only a second backbone used for robustness checks.
Experimental setup
Datasets: Ego4D
Benchmarks: EgoBench
Metrics: Joint Success Rate ToolSucc MicroAcc ResultSucc All Success (simulated user)
Baselines: Qwen3-VL-235B Qwen3.6-Plus Kimi-K2.6 MiMo-V2.5-Omni GLM-5V-Turbo Doubao-Seed-2.0-Pro Qwen3.5-397B-A17B
Builds on: tau-bench tau^2-bench ToolSandbox BFCL Ego4D
Closest papers by meaning
- Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos (CVPR 2026 · similarity 0.91)
- VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents (ICLR 2025 · similarity 0.89)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.89)
- AgenticVBench: Can AI Agents Complete Real-World Video Production Tasks? (NeurIPS 2026 · similarity 0.89)
- AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.88)
- Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.