Research map / Vision-language and multimodal / Multimodal LLMs

DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DocHop, a synthetic benchmark of 2,074 single-page document images pairing narrative multi-hop constraints with multiple charts, and evaluates proprietary and open-source MLLMs against human annotators.

Problem

Existing chart and document QA benchmarks evaluate visual structured understanding in isolation and rarely test whether models use textual narrative constraints to select, interpret, and aggregate chart evidence across multiple charts.

Approach

A logic-first reverse-engineering pipeline samples a symbolic reasoning trace, generates chart tables and a narrative verbalizing the trace independently, and grounds questions on a semantic reference label defined in the narrative, with controllable reasoning depth and chart count.

Key results

• GPT-5.2 Reasoning achieves the best overall accuracy of 62.83% on DocHop, versus about 93.29% for human annotators.
• Reasoning-enhanced variants outperform non-reasoning counterparts, e.g. GPT-5.2 rises from 40.36% to 62.83% overall.
• Open-source models mostly score roughly 9-24% overall, with Qwen-2.5-VL-72B at 24.40%.

Surprising findings

• Question-only guessing baselines reach up to about 11% overall, and near 46-59% on fact checking, indicating a strong language prior on binary questions.

Limitations

• Synthetic, controlled single-page A4 layouts do not capture the visual variability of real documents.
• Only chart-based numerical evidence is covered; tables, diagrams and UI screenshots are not included.
• Synthetic generation may leave occasional annotation, wording, or rendering issues.

Experimental setup

Benchmarks: DocHop

Metrics: normalized answer accuracy

Baselines: GPT-5.2 GPT-5-mini Gemini-2.5-Pro Gemini-2.5-Flash Claude-4.5-Haiku Claude-4.5-Sonnet LLaVA-Next-LLaMA3-8B IDEFICS2-8B IDEFICS3-LLaMA3-8B Qwen-2.5-VL-7B Qwen3-VL-8B Molmo-7B-O-0924

Builds on: ChartQA ChartXiv

Closest papers by meaning

• DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding (CVPR 2026 · similarity 0.89)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning (NeurIPS 2026 · similarity 0.88)
• MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.88)
• DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding (NeurIPS 2026 · similarity 0.88)
• Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.88)