Research map / Multi-agent systems and games / LLM agents

AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AgentVista, a 209-task benchmark of realistic, visually detailed multimodal agent tasks requiring long-horizon interleaved tool use (web search, image search, visit, code interpreter), where the best model reaches 27.3% accuracy.

Problem

Existing multimodal benchmarks evaluate single-turn visual reasoning or narrow tool skills and either simplify visual inputs or use shallow tool chains, so they do not measure realistic long-horizon multimodal agent workflows.

Approach

A manually curated benchmark of realistic, detail-rich images and user requests (single and multi-image) with verifiable short answers, filtered so that solving requires interleaved calls across at least two tool categories and cannot be solved without tools or from text alone.

Key results

• Gemini-3-Pro with tools achieves the best overall accuracy of 27.27% on AgentVista, while open-source models range from 10.05% to 12.92%.
• Gemini-3-Flash test-time scaling: best-of-16 with a Gemini-3-Flash reward model reaches 30.62%, and pass@16 reaches 51.67%, versus 21.05% at K=1.
• Tool ablation on Gemini-3-Pro: full tools 27.27%, vision-only 20.10%, no tool 18.18%; search-only 26.32%.

Surprising findings

• Multi-image inputs yield higher accuracy than single-image inputs for nearly all models (e.g., Gemini-3-Pro 36.84% vs 23.68%).
• Visual misidentification is the dominant failure mode across models, and frontier models often fail even after zooming in on relevant regions.

Limitations

• Small benchmark of 209 tasks, so per-category results rest on few instances.
• Evaluation uses a fixed four-tool environment and a 30-turn budget, which may not reflect other agent setups.
• Error labels are generated automatically by Gemini-3-Flash rather than by humans.

Experimental setup

Datasets: VisionArena WildVision

Benchmarks: AgentVista

Metrics: accuracy average tool-calling turns pass@k best-of-k

Baselines: GPT-4.1 o3 o4-mini GPT-5 GPT-5.1 GPT-5.2 Grok-4 Claude-Sonnet-4 Claude-Opus-4.1 Claude-Sonnet-4.5 Gemini-3-Flash Gemini-3-Pro

Builds on: VisualToolBench BrowseComp-VL MMSearch-Plus WildVision VisionArena

Closest papers by meaning

• Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.91)
• VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents (ICLR 2025 · similarity 0.91)
• DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.90)
• InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search (ICLR 2026 · similarity 0.89)
• Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis (NeurIPS 2026 · similarity 0.89)
• VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks (ICLR 2025 · similarity 0.89)