Research map / Vision-language and multimodal / Multimodal LLMs

DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DocScope, a 1,124-question benchmark for verifiable long-document QA where MLLMs output evidence pages, bounding-box regions, facts and answers, scored by a four-stage trajectory protocol with LLM judges.

Problem

Answer accuracy alone does not reveal whether multimodal LLMs give trustworthy, verifiable reasoning over long, visually rich PDF documents; existing benchmarks use fixed evidence pools or lack region- and fact-level evidence supervision.

Approach

Formulate long-document QA as structured trajectory prediction (pages, normalized bounding boxes, factual statements, answer) and audit each stage independently with decoupled metrics, using human-annotated hierarchical evidence and LLM judges calibrated against humans.

Key results

• Gemini 3.1 Pro achieves the highest answer accuracy at 78.9%, but only 39.7% strict region F1 and 68.7% fact consistency.
• Among answer-correct samples, the highest complete evidence-chain rate (GPT-5.4) is only 29.06%.
• Oracle study shows faithful fact extraction is the dominant bottleneck; even with oracle facts Ministral3-8B reaches 78.6% vs Claude Sonnet 4.6 at 92.0%.

Surprising findings

• Even among correct answers, the highest complete evidence-chain rate is only 29%, showing answer accuracy overestimates trustworthiness.
• Dense models outperform their MoE counterparts within families despite fewer total parameters, suggesting activated parameters matter more than total scale.

Limitations

• Annotation completeness is hard to guarantee; a unique minimal evidence set is difficult to establish (2.67% of sampled instances showed required missing evidence).
• Questions are synthesized by MLLMs (Claude-Opus-4.6), and the benchmark is English-only with documents from a single crawl source.
• Judges are LLMs (GPT-5.5, Qwen3.6-Plus) whose reliability depends on human alignment studies.

Experimental setup

Datasets: FinePDF

Benchmarks: DocScope

Metrics: page precision page recall page F1 strict region F1 lenient region F1 fact consistency answer accuracy evidence-chain completeness

Baselines: MMLongBench-Doc pipeline SimpleDoc VidoRAG URaG Docopilot

Builds on: MMLongBench-Doc LongDocURL

Closest papers by meaning

• DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding (CVPR 2026 · similarity 0.89)
• DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA (ICML 2026 · similarity 0.89)
• DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents (ICML 2026 · similarity 0.88)
• TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning (NeurIPS 2026 · similarity 0.88)
• CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.87)
• MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.87)