Research map / Vision-language and multimodal / Multimodal LLMs

Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ScholScan, a 1,800-question benchmark for scan-oriented error detection across full academic papers, evaluating 15 models in 24 input configurations plus RAG methods, finding low scores and no RAG gains.

Problem

Existing academic paper reasoning benchmarks follow a search-oriented, QA paradigm with pre-specified targets and local evidence, which does not test whether models can scan entire papers and proactively detect non-obvious consistency errors.

Approach

Reframes paper reasoning as scan-oriented: models get complete papers with target-absent queries and must find injected or reviewer-sourced scientific errors, evaluated with a process-aware score that combines detection, evidence localization, reasoning-chain prefix match, and an unrelated-error penalty.

Key results

• GPT-5 achieves the highest average image-input score of 19.2; Gemini 2.5 Pro reaches 30.3 average with text (OCR) input, the best overall setting.
• Across the board RAG methods yield no significant improvements; the best RAG method VRAG-RL reaches 10.9 average versus 7.0 baseline for Llama 4 Maverick with image input.
• Text input outperforms image input by 4.81 points on average across nine MLLMs.

Surprising findings

• Retrieval-augmented generation with embedding models often degrades performance after retrieval, with multimodal embedding models approaching zero scores.
• Oracle gold-image access gives only limited gains, and even the strongest models scan up to 8 times more evidence than the reference.

Limitations

• Errors are largely synthetically injected by Gemini 2.5 Pro, so they may not reflect naturally occurring errors.
• Evaluation relies on GPT-4.1 as extractor/judge, with human validation only partially reported.

Experimental setup

Datasets: ICLR 2024 papers ICLR 2025 papers Nature Communications papers ICLR public reviews

Benchmarks: ScholScan

Metrics: overall score S(a) combining detection, evidence location Di MRR@5 Recall@5

Baselines: GPT-5 Gemini 2.5 Pro Doubao-Seed-1.6 Grok 4 Llama 4 Maverick Qwen3-Thinking DeepSeek-R1 BM25 Contriever BGE-M3 NV-Embed-v2 VRAG-RL

Builds on: MMLongBench-Doc VRAG-RL VisRAG

Closest papers by meaning

• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding (CVPR 2026 · similarity 0.87)
• Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections (ICML 2026 · similarity 0.87)
• TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning (NeurIPS 2026 · similarity 0.86)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.86)
• SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.86)
• DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding (NeurIPS 2026 · similarity 0.86)