Research map / Vision-language and multimodal / Multimodal LLMs

MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MET-Bench, a multimodal entity tracking benchmark (Chess, Shell Game, Minecraft) showing VLMs track entity states much worse from images than text, and that GRPO improves in-modality tracking without robust cross-modal transfer.

Problem

Vision-language models struggle to track entity state when updates are conveyed as images rather than text, and prior entity tracking benchmarks are text-only.

Approach

A parallel text-vs-image benchmark where the same state-tracking task is posed with text or image actions/states, isolating visual reasoning from perception via a cascaded (image-to-text then track) inference test, plus GRPO training per modality.

Key results

• Claude 3.7 Sonnet reaches 96.1% text vs 70.2% image per-square accuracy in zero-shot Chess; Claude 4.5 Opus and GPT-5.2 reach ~99% on image Chess with reasoning.
• GRPO on text-only Shell Game lifts Gemma 3 4B IT from 37.6% to 92.2% in text, but image accuracy drops to 27.4%, showing weak cross-modal transfer.
• Cascaded inference (image-to-text then tracking) recovers near-text performance in Shell Game, indicating visual reasoning rather than perception is the bottleneck.

Surprising findings

• Shell Game image perception is near-perfect for most models, yet image-based tracking collapses to near random without chain-of-thought.
• GRPO training on one modality does not improve, and can hurt, the held-out modality (e.g., GRPO Text drops Shell Game image accuracy to 27.4%).

Limitations

• Evaluated on controlled game domains (Chess, Shell Game, Minecraft) rather than real-world environments.
• Only Gemma 3 4B IT trained with RL; limited open-source model coverage for training experiments.
• Chess image-action setting required SFT warm-start on synthetic demonstrations because base policy lacked valid reasoning traces.

Experimental setup

Datasets: Millionbase chess games MET-Bench Chess MET-Bench Shell Game MET-Bench Minecraft

Benchmarks: MET-Bench Chess MET-Bench Shell Game MET-Bench Minecraft

Metrics: per-square accuracy exact-match FEN accuracy changed-square accuracy legal board rate final-position accuracy multiple-choice accuracy perception accuracy

Baselines: Random Game Start baseline GPT-4o-mini Gemma 3 27B IT

Builds on: GRPO TRL

Closest papers by meaning

• Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval (NeurIPS 2025 · similarity 0.87)
• SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.87)
• MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.86)
• ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model (NeurIPS 2024 · similarity 0.86)
• JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.86)
• ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.86)