Research map / Vision-language and multimodal / Multimodal LLMs
The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks
Shuo Wen, Beixi Du, Edwin Meriaux, Junming Shi, Chloe Si, David Meger, Doina Precup, Gregory Dudek
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces The Prestige, a benchmark of magic-trick videos for evaluating cognitive visual reasoning in vision-language models, aiming to avoid multiple-choice shortcuts.
Problem
Existing video understanding benchmarks often overlook deep cognitive processes in visual reasoning and rely on multiple-choice formats that encourage reasoning shortcuts.
Approach
Uses magic tricks as video material to probe cognitive visual reasoning from visual information alone, moving away from multiple-choice formats.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide 4 candidate answers (4-way MC) instead of open-ended answering (ablation) | Qwen3.6-Plus, Gemini3.1-Flash-Lite, Kimi-K2.6 · The Prestige, 200 random questions | mean accuracy (%): 37.5 -> 71.5 | Improved |
| provide 8 candidate answers (8-way MC) instead of open-ended answering (ablation) | Qwen3.6-Plus, Gemini3.1-Flash-Lite, Kimi-K2.6 · The Prestige, 200 random questions | mean accuracy (%): 37.5 -> 67.2 | Improved |
| provide MC options on questions all three models failed open-ended (ablation) | Qwen3.6-Plus, Kimi-K2.6, Gemini3.1-Flash-Lite · The Prestige, 50 failure-selected questions | accuracy (%): 0% -> 60-78% (4-way mean gain 73.3 points) | Improved |
| fewer distractors (4-way instead of 8-way MC) (ablation) | Qwen3.6-Plus, Kimi-K2.6, Gemini3.1-Flash-Lite · The Prestige, 200 random questions | mean accuracy (%): 67.2 -> 71.5 | Improved |
| score with majority-vote LLM graders instead of human graders | 14 VLMs · The Prestige, overall | overall accuracy gap (points): human minus machine mean 4.56 (SD 1.23) | Hurt |
| scale up model size / use larger open-source models | open-source VLMs · The Prestige, overall | overall accuracy (%): 14.15-29.12 (small) -> 38.26-39.37 (Qwen3.6, Kimi-K2.6) | Improved |
| thinking mode instead of instruct mode | Qwen3VL-8B · The Prestige, overall | overall accuracy (%): 29.12 -> 23.84 | Hurt |
| question type PR (explain why a reveal is surprising) versus IR | Qwen3VL-Thinking-8B · The Prestige | accuracy (%): IR 40.00 -> PR 9.76 | Hurt |
Experimental setup
Benchmarks: The Prestige
Closest papers by meaning
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.90)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.90)
- When Thinking Drifts: Evidential Grounding for Robust Video Reasoning (NeurIPS 2025 · similarity 0.89)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.89)
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.89)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
