Research map / Vision-language and multimodal / Multimodal LLMs

Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Agent-X, a 828-task benchmark of vision-centric agentic tasks with real visual inputs and tool chains, plus step-level metrics for deep reasoning, and evaluates ~10 LMMs finding none exceed 50% goal accuracy.

Problem

Existing agentic benchmarks use synthetic single-turn queries, limited visual modalities, and lack step-level metrics for multi-step multimodal reasoning and tool use in vision-centric settings.

Approach

A semi-automated benchmark of 828 tasks built from real visual inputs (images, multi-image, video) with implicit-tool queries, paired with executable 14-tool chains, and evaluated via fine-grained step-by-step, deep-reasoning, and outcome metrics using LMM judges plus human checks.

Key results

• No model exceeds 50% Goal Accuracy; o4-mini is best at 0.45, while most open-source models are below 0.30.
• GPT-4o achieves the highest Faithfulness (0.81) and Factual Precision (0.79) but a Toolset Accuracy of 0.68 and Goal Accuracy of 0.37.
• Qwen2.5-VL-7B reaches Goal Accuracy 0.36 and Tool Accuracy 0.63, competitive with closed-source models.

Surprising findings

• GPT-4o scores high on reasoning metrics yet its toolset accuracy is low, suggesting tool invocation is a bottleneck separate from reasoning.
• Gemini-1.5-Pro has very low Goal Accuracy (0.04) despite high Tool Accuracy (0.84) in the table.

Limitations

• Monolingual benchmark that may inherit distributional bias.
• Semi-automated GPT-4o query and toolchain generation can yield lower-quality samples.
• Limited scale relative to the stated goal of broad coverage.

Experimental setup

Datasets: Cityscapes BDD100K Nighttime Driving nuScenes Surveillance Camera Fight Dataset AI City Challenge 2023 MathVista AI2D Mobile Interfaces WebArena SoccerNet Sports-1M

Benchmarks: Agent-X

Metrics: Grounding Score Tool Precision Tool Accuracy Faithfulness Accuracy Context Score Factual Precision Semantic Accuracy Goal Accuracy Goal Accuracy w/ImgGen Toolset Accuracy

Baselines: GPT-4o Gemini-1.5-Pro Gemini-2.5-Pro OpenAI-o4-mini Qwen2.5-VL-7B InternVL2.5-8B InternVL3-8B VideoLLaMA3-7B Phi-4-VL-Instruct Gemma-3-4B Pixtral-12B LLaMA-3.2-11B-Vision

Builds on: GTA m&m's ToolBench GAIA

Closest papers by meaning

• AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.91)
• GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning (NeurIPS 2025 · similarity 0.91)
• Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning (ICML 2026 · similarity 0.91)
• DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.90)
• ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both (NeurIPS 2026 · similarity 0.90)
• CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization (CVPR 2026 · similarity 0.90)