Research map / Vision-language and multimodal / Multimodal LLMs

GameVerse: Can Vision-Language Models Learn from Video-based Reflection?

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GameVerse, a 15-game video benchmark with a cognitive taxonomy, dual action space, milestone scoring, and a reflect-and-retry paradigm where VLMs learn from failure trajectories and expert tutorial videos.

Problem

Existing VLM game benchmarks use fire-and-forget evaluation, rely on privileged text state or coarse genre taxonomies, and do not test whether agents can learn from failures and expert video tutorials.

Approach

A reflect-and-retry loop where the VLM contrasts its failed trajectory with a retrieved expert walkthrough video, writes textual reflections injected into the system prompt for the next attempt; plus a cognitive taxonomy (image structure, temporal dynamics, causal linearity) and VLM-based milestone scoring from pixels.

Key results

• Gemini-2.5-Pro improves by 6.47% with video-based reflection versus 1.60% for GPT-4o-mini, and gains shrink from 4.4% in non-real-time to 1.7% in real-time games.
• Combining failure reflection and tutorials (Self) gives the best results: GPT-4o-mini averages 65.5 vs 45.6 without reflection on three semantic-mode games; Qwen3-VL-32B reaches 76.7 vs 61.0.
• Human experts average 100 on most games while the best model (Gemini-2.5-Pro, average rank 7.5) collapses to 0 on several hard games such as Civilization VI, Scene Investigators and RDR 2.

Surprising findings

• Reflections generated by the stronger Qwen3-VL-32B do not help the weaker GPT-4o-mini more than its own reflections, suggesting internalization rather than extraction is the bottleneck.
• Reasoning models (Gemini-2.5-Pro, Seed-1.8) degrade sharply under real-time latency while reactive models remain stable.

Limitations

• Reflection is passive and single-turn; agents cannot query specific video segments or iterate.
• Milestone scoring relies on a single VLM (Gemini-3-pro) and discrete frame sampling lowers matching scores for agent videos.
• Evaluation limited to 3-20 trials per game due to cost, and human baselines are small (37 participants).

Experimental setup

Datasets: YouTube/Bilibili expert walkthrough and tutorial videos (15 games)

Benchmarks: GameVerse

Metrics: normalized game score (0-100) milestone completion ratio

Baselines: Random Human Rookie Human Expert GPT-4o GPT-4o-mini Gemini-2.5-Pro Gemini-2.5-Flash Seed-1.8

Builds on: Voyager UI-TARS FlashAdventure Reflexion

Closest papers by meaning

• Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play (ICLR 2026 · similarity 0.90)
• Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.90)
• Are Large Vision Language Models Good Game Players? (ICLR 2025 · similarity 0.88)
• VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.88)
• BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.87)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.87)