Research map / Vision-language and multimodal / Multimodal LLMs

PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PRISM-Bench, a 1044-item puzzle benchmark with a first-error detection track over single-error chain-of-thought, evaluated on frontier and open MLLMs.

Problem

Existing multimodal reasoning benchmarks score only final answers, so they cannot tell whether MLLMs' step-by-step reasoning is faithful or where it first goes wrong.

Approach

Pairs each visual puzzle with a ground-truth CoT and a corrupted CoT containing exactly one injected error at a known step (later steps rewritten for coherence), so models must identify the first incorrect step, separately from final-answer accuracy.

Key results

• First-error detection: SkyWork-R1V3-38B reaches 62.3% overall, MiniCPM-V-4.5 58.1%, Qwen2.5-VL-7B 57.0%, while several open models (e.g., YiVL-34B 12.3%) are near or below chance.
• VQA puzzle solving: GPT-5 achieves 39.6% overall accuracy, the best result, with most other models below 28%.
• Macro-average VQA accuracy and first-error detection accuracy are only moderately correlated (Spearman rho = 0.62, Kendall tau = 0.47).

Surprising findings

• Models often produce fluent, plausible CoTs yet fail to locate simple injected logical errors, with some models blaming correct earlier steps (back-propagated blame).
• Reasoning-first prompting yields only marginal accuracy differences versus direct answering while substantially increasing inference time.

Limitations

• Corrupted CoTs were generated by GPT-o3 and reviewed by annotators, so generator-specific artifacts may exist.
• Puzzles come from a single exercise book, limiting diversity of puzzle sources.
• Error detection requires a single injected error per CoT, which may not reflect naturally occurring multi-error reasoning.

Experimental setup

Datasets: Exercise book of visual puzzles (over 16k raw puzzles, filtered to 1044)

Benchmarks: PRISM-Bench

Metrics: answer accuracy first-error detection accuracy Spearman correlation Kendall tau

Baselines: GPT-5 GPT-o3 SkyWork-R1V3-38B MiniCPM-V-4.5 Qwen2.5-VL-7B-Instruct InternVL-2.5-78B VL-Rethinker-7B MiMo-VL-7B-RL-2508

Builds on: Chain-of-thought prompting PRM800K

Closest papers by meaning

• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.92)
• PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts (ICLR 2026 · similarity 0.91)
• Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.91)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.90)
• StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning (NeurIPS 2026 · similarity 0.90)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.90)