AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks

Shuo Wen, Beixi Du, Edwin Meriaux, Junming Shi, Chloe Si, David Meger, Doina Precup, Gregory Dudek

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces The Prestige, a benchmark of magic-trick videos for evaluating cognitive visual reasoning in vision-language models, aiming to avoid multiple-choice shortcuts.

Problem

Existing video understanding benchmarks often overlook deep cognitive processes in visual reasoning and rely on multiple-choice formats that encourage reasoning shortcuts.

Approach

Uses magic tricks as video material to probe cognitive visual reasoning from visual information alone, moving away from multiple-choice formats.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
provide 4 candidate answers (4-way MC) instead of open-ended answering (ablation)Qwen3.6-Plus, Gemini3.1-Flash-Lite, Kimi-K2.6 · The Prestige, 200 random questionsmean accuracy (%): 37.5 -> 71.5Improved
provide 8 candidate answers (8-way MC) instead of open-ended answering (ablation)Qwen3.6-Plus, Gemini3.1-Flash-Lite, Kimi-K2.6 · The Prestige, 200 random questionsmean accuracy (%): 37.5 -> 67.2Improved
provide MC options on questions all three models failed open-ended (ablation)Qwen3.6-Plus, Kimi-K2.6, Gemini3.1-Flash-Lite · The Prestige, 50 failure-selected questionsaccuracy (%): 0% -> 60-78% (4-way mean gain 73.3 points)Improved
fewer distractors (4-way instead of 8-way MC) (ablation)Qwen3.6-Plus, Kimi-K2.6, Gemini3.1-Flash-Lite · The Prestige, 200 random questionsmean accuracy (%): 67.2 -> 71.5Improved
score with majority-vote LLM graders instead of human graders14 VLMs · The Prestige, overalloverall accuracy gap (points): human minus machine mean 4.56 (SD 1.23)Hurt
scale up model size / use larger open-source modelsopen-source VLMs · The Prestige, overalloverall accuracy (%): 14.15-29.12 (small) -> 38.26-39.37 (Qwen3.6, Kimi-K2.6)Improved
thinking mode instead of instruct modeQwen3VL-8B · The Prestige, overalloverall accuracy (%): 29.12 -> 23.84Hurt
question type PR (explain why a reveal is surprising) versus IRQwen3VL-Thinking-8B · The Prestigeaccuracy (%): IR 40.00 -> PR 9.76Hurt

Experimental setup

Benchmarks: The Prestige

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy