Research map / Vision-language and multimodal / Multimodal LLMs
VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoGameQA-Bench, a benchmark of 3,436 image and video samples across nine video game QA tasks, and evaluates 16 VLMs, finding strong glitch detection but weak fine-detail perception, visual regression, and temporal localization.
Problem
Existing multimodal benchmarks do not measure the visual perception and commonsense skills that video game quality assurance needs, such as unit testing, regression testing, glitch detection, and bug reporting, so VLM readiness for game QA is unknown.
Approach
A benchmark that maps realistic game QA workflows (visual/UI unit tests, regression, glitch and clipping detection, needle-in-a-haystack glitch localization, bug report generation) onto image and video questions with JSON-structured answers, using real Steam/YouTube/GamePhysics data plus Unity-synthesized controlled samples.
Key results
- GPT-4.1 reaches 81.3% accuracy on image glitch detection and 87.8% on parametric clipping detection, while Gemini 2.5 Pro reaches 78.1% on video glitch detection.
- Visual regression is hard: the best model, o4-mini, reaches only 45.2% accuracy.
- In needle-in-a-haystack, the best models (Gemini 2.5 Flash variants) locate the glitch frame within 5 seconds only 35–36% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use reasoning-model variant (o3) instead of non-reasoning GPT-4o | visual regression testing · VideoGameQA-Bench visual regression | accuracy (%): 39.6 vs 31.6 | Helped |
| use reasoning variant (new Claude 3.7 Sonnet) instead of Claude 3.5 Sonnet | visual regression testing · VideoGameQA-Bench visual regression | accuracy (%): 24.0 vs 14.0 | Helped |
| sample video at 1 FPS and pass frames to non-Gemini models | video glitch detection · VideoGameQA-Bench video | accuracy (%): proprietary models generally perform slightly worse on video than image | Hurt |
| prompt-wording variation (10 alternative prompts) for image glitch detection on GPT-4.1 (ablation) | image glitch detection · VideoGameQA-Bench image | accuracy (%): mean 80.12 vs 81.3 base; std 3.25 | No clear effect |
| parse-error and refusal handling: GPT-4o refusals on video tasks | video NIAH and video glitch detection · VideoGameQA-Bench NIAH | parse-error rate (%): 93.0 parse error on NIAH | Hurt |
| Gemini-family models accept native video input rather than sampled frames | video tasks · VideoGameQA-Bench video | accuracy (%) | Mixed |
Surprising findings
- Reasoning variants outperform non-reasoning counterparts on visual regression testing but not on glitch detection.
- GPT-4o refuses or fails on most video NIAH queries (93% parse-error rate), so its video scores are not comparable.
Limitations
- Benchmark samples focus on post-release glitches; development-time bugs are not covered.
- LLM-as-a-judge introduces an estimated ~5% judging error for bug reports.
- Video is processed as sampled frames at 1 FPS, losing temporal context and audio, and open-weight models use lower sampling rates.
Experimental setup
Datasets: Steam Community screenshots GamePhysics YouTube gameplay videos Unity synthetic scenes
Benchmarks: VideoGameQA-Bench
Metrics: accuracy precision recall specificity F1 parse-error rate
Baselines: GlitchBench PhysBench
Builds on: GlitchBench GamePhysics dataset needle-in-a-haystack retrieval
Closest papers by meaning
- Are Large Vision Language Models Good Game Players? (ICLR 2025 · similarity 0.86)
- GlitchBench: Can Large Multimodal Models Detect Video Game Glitches? (CVPR 2024 · similarity 0.86)
- GameVerse: Can Vision-Language Models Learn from Video-based Reflection? (ICML 2026 · similarity 0.85)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.85)
- Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.85)
- R4-CGQA: Retrieval-based Vision Language Models for Computer Graphics Image Quality Assessment (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.