Research map / Multi-agent systems and games / LLM agents

VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VS-Bench, a multimodal benchmark of ten vision-grounded multi-agent games evaluating 15 VLMs on perception, strategic reasoning (next-action prediction), and decision-making (normalized return).

Problem

Existing VLM agent benchmarks focus on single-agent or text-only settings, leaving strategic abilities of VLMs in multimodal multi-agent environments (cooperative, competitive, mixed-motive) unmeasured.

Approach

Adapts ten classic MARL and game-theory environments (Hanabi, Overcooked, KAZ, Breakthrough, Kuhn Poker, Atari Pong, MPE, Coin Dilemma, Monster Hunt, Battle of the Colors) to image+text observations and evaluates VLMs along three dimensions: perception accuracy, next-action prediction accuracy, and normalized episode return.

Key results

• Best model o3 reaches 46.6% next-action prediction accuracy overall, versus oracle 100%.
• Best model o3 attains 31.4% normalized episode return; human participants average 62.7.
• Perception is strong: all models at least 67.8% overall accuracy, best o3 at 84.9%.

Surprising findings

• Four of fifteen models score worse than random agents on overall decision-making return.
• Text-only observations yield only slightly better decision-making than multimodal observations, indicating the difficulty lies in strategic interaction rather than visual perception alone.

Limitations

• Evaluation limited to ten games and 400 samples per environment for perception and reasoning.
• Models evaluated only in zero-shot prompting setups without training.
• Decision-making results depend on sampled episodes and self-play or conventional-agent opponents.

Experimental setup

Datasets: Hanabi Overcooked KAZ Breakthrough Kuhn Poker Atari Pong MPE Coin Dilemma Monster Hunt Battle of the Colors

Benchmarks: VS-Bench

Metrics: element recognition accuracy next-action prediction accuracy normalized episode return

Baselines: o3 claude-3-7-sonnet gemini-2.5-pro ui-tars-1-5 doubao-1-5-thinking-pro qvq-max gpt-4.1 Qwen2.5-VL-72B-Ins.

Builds on: Partially Observable Markov Games Chain-of-Thought prompting

Closest papers by meaning

• BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.90)
• Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning (NeurIPS 2025 · similarity 0.89)
• Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs (CVPR 2026 · similarity 0.89)
• ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting (CVPR 2025 · similarity 0.89)
• VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents (ICLR 2025 · similarity 0.88)