Research map / Vision-language and multimodal / Multimodal LLMs

GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GlitchBench, a free-text benchmark of 593 game glitch images (plus 330 glitch-free) to test whether large multimodal models can detect unusual events, and evaluates 11 LMMs with a Llama-2 judge.

Problem

Existing LMM benchmarks use yes/no or multiple-choice formats and do not test whether models can detect out-of-the-ordinary real-world anomalies such as video game glitches, requiring visual perception and commonsense/physics reasoning.

Approach

Builds a benchmark from real-world game quality-assurance data (Reddit GamePhysics glitch frames plus Unity-synthesized glitches) and asks open-ended questions with no linguistic hints, using an LLM judge to score semantic agreement with a one-line ground-truth description.

Key results

• GPT-4V achieves the best average accuracy on GlitchBench of 43.4% (Q1 57.2%, Q2 29.5%), while the best open-source model LLaVA-1.5-13B reaches 35.5% average.
• GPT-4V reaches 64.9% under the extensive captioning setting (Q3), versus 43.4% on the main benchmark.
• Fuyu-8B scores only 8.5% on GlitchBench despite exceeding GPT-4V on OKVQA and AI2D, showing weak correlation between existing benchmarks and real-world glitch detection.

Measured effects reported in the paper

Surprising findings

• Open-source models often match or surpass GPT-4V on standard benchmarks like VQAv2 and OKVQA yet fall far behind on GlitchBench.
• Providing three temporal frames to GPT-4V reduced accuracy (Q1 39% to 36%, Q2 35% to 28%).

Limitations

• Dataset sampled from Reddit leans heavily toward open-world games.
• Llama-2 judge agreement with humans varies by model (Cohen's kappa from -0.09 for Fuyu to 1.00 for Qwen-VL) and was checked on only 220 samples.
• Only single-frame glitches are included; temporal glitches are excluded by design.

Experimental setup

Datasets: GamePhysics GlitchBench

Benchmarks: GlitchBench VQAv2 OKVQA AI2D SEED POPE MMBench

Metrics: accuracy (judge-determined) Cohen's kappa

Baselines: LLaVA-1.5 SPHINX InstructBLIP Qwen-VL MiniGPT-v2 OtterHD Fuyu

Builds on: GamePhysics Whoops

Closest papers by meaning

• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.87)
• Are Large Vision Language Models Good Game Players? (ICLR 2025 · similarity 0.86)
• VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance (NeurIPS 2025 · similarity 0.86)
• Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.86)
• WhodunitBench: Evaluating Large Multimodal Agents via Murder Mystery Games (NeurIPS 2024 · similarity 0.86)
• SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.85)