Research map / Vision-language and multimodal / Multimodal LLMs

VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VipAct, a VLM agent framework where an orchestrator GPT-4o coordinates specialized VLM agents (captioning, visual prompt description, image comparison) and vision expert tools (depth, detection, segmentation, similarity) to improve fine-grained visual perception on Blink and MMVP.

Problem

VLMs struggle with fine-grained visual perception requiring pixel-level analysis, such as depth, correspondence, and visual prompt understanding, and existing text prompting, visual prompting, and visual programming methods do not reliably help.

Approach

A VLM orchestrator agent plans, calls specialized VLM sub-agents and vision expert models via function calling, and receives returned images as well as text, so visual outputs re-enter the reasoning loop; it is prompted with one template across tasks.

Key results

• On Blink with GPT-4o, VipAct averages 73.79% versus 65.85% for the best text-prompting baseline (CoT) and 63.74% zero-shot.
• On MMVP with GPT-4o, VipAct reaches 70.7% versus 68.0% for zero-shot and 66.0% for LtM/ToT.
• Ablation on Blink shows removing direct image input to the orchestrator drops MMVP accuracy from 70.7% to 54.0%.

Surprising findings

• Visual programming baselines (MM-ReAct, ViperGPT, VisProg) often perform far below zero-shot, e.g., MM-ReAct at 6.67% on MMVP.
• Text-based CoT/LtM/ToT do not consistently improve over zero-shot, and SoM impairs performance on most tasks.

Limitations

• Open-source VLMs often fail to follow the required output format, limiting applicability beyond GPT-4o, Gemini and Claude.
• Error analysis shows persistent failures in small-part perception, closely spaced visual prompts, and spatial reasoning from object perspective.
• Evaluated only on two benchmarks with small per-subtask sizes (about 120-170 items).

Experimental setup

Datasets: Blink MMVP

Benchmarks: Blink MMVP

Metrics: accuracy

Baselines: Zero-shot CoT Least-to-most prompting Tree-of-thought Set-of-Mark MM-ReAct ViperGPT VisProg

Builds on: ReAct MM-ReAct ViperGPT VisProg Set-of-Mark

Closest papers by meaning

• GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning (NeurIPS 2025 · similarity 0.90)
• Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.89)
• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
• Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs (CVPR 2026 · similarity 0.88)
• VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.88)