Research map / Vision-language and multimodal / Multimodal LLMs

PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds PRISMM-Bench, a 384-item benchmark of real reviewer-flagged multimodal inconsistencies from ICLR 2024/2025 papers, with three MCQ tasks and JSON-structured answers to curb choice-only shortcuts, and evaluates 21 LMMs.

Problem

Whether large multimodal models can detect, remedy and reason over real inconsistencies between text, figures, tables and equations in scientific papers; existing benchmards use single modalities or synthetic errors, and MCQ evaluation suffers from choice-only linguistic shortcuts.

Approach

Mine reviewer-flagged inconsistencies from rejected/withdrawn OpenReview submissions using LLM filtering plus manual verification, turn them into three MCQ tasks, and convert answer options into structured Evidence-Claim / Target-Action JSON to suppress linguistic shortcuts.

Key results

• GPT-5 (high reasoning) achieves the highest average accuracy of 53.9% across tasks and contexts, while the best open-weight model GLM 4.5V 106B reaches 42.6%.
• Without context, Gemini 2.5 Flash reaches 57.6% on Ident with natural-language options, dropping to 34.0% after JSON conversion; under JSON, model no-context accuracy approaches chance.
• In the user study, humans scored 27.5% without context and 77.5% with focused context, while Gemini 2.5 Pro scored 81.6% focused and 83.8% with whole document context.

Surprising findings

• Gemma 3 12B reaches 63.5% on Match, rivaling proprietary models, while the much larger InternVL3 78B scores 45.3%.
• Higher rasterization resolution (300 DPI) does not consistently improve accuracy; it hurts several InternVL models.

Limitations

• Benchmark is restricted to AI-domain ICLR 2024/2025 papers and rejected or withdrawn submissions, limiting scale and domain coverage.
• Annotation was done by a single author, and the JSON debiasing was only manually verified on a 20% subset.

Experimental setup

Datasets: ICLR 2024 OpenReview reviews ICLR 2025 OpenReview reviews

Benchmarks: PRISMM-Bench Ident PRISMM-Bench Remedy PRISMM-Bench Match

Metrics: accuracy visual reliance ratio

Baselines: GPT-5 Gemini 2.5 Pro InternVL3 78B GLM 4.5V 106B Qwen 2.5 VL 72B LLaVA OneVision 72B Gemma 3 27B Ovis2 34B

Builds on: MMLongBench-Doc MMIR SPIQA

Closest papers by meaning

• Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts (AAAI 2026 · similarity 0.90)
• MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research (CVPR 2025 · similarity 0.87)
• SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks (NeurIPS 2025 · similarity 0.87)
• MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
• Trust, but Don’t Verify: Epistemic Blind Spots in LLM Source Evaluation (NeurIPS 2026 · similarity 0.87)
• From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.87)