Research map / Vision-language and multimodal / Multimodal LLMs

MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 22 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMLONGBENCH-DOC, a 1,082-question benchmark over 135 lengthy PDFs to evaluate long-context multimodal document understanding of LVLMs versus OCR+LLM pipelines.

Problem

Long-context document understanding (tens of pages, multimodal evidence, cross-page reasoning, unanswerable questions) is not covered by existing single-page or short multi-page DU benchmarks, and LVLM capabilities there are unclear.

Approach

A benchmark of lengthy, multi-modal PDFs with expert-annotated questions labeled by evidence source, evidence page count (single-page, cross-page, unanswerable), and answer format, evaluated end-to-end on page screenshots and compared against Tesseract OCR-parsed text fed to LLMs.

Key results

• GPT-4o achieves the best overall F1 of 44.9% on MMLongBench-Doc, with GPT-4V second at 31.2% F1.
• Human experts reach 66.0% F1, exceeding the best model by more than 20 points.
• 12 of 14 LVLMs score below their OCR-parsed text LLM counterparts; the best open-source LVLM trails the best open-source LLM (Mixtral-8x22B) by 11.7%.

Measured effects reported in the paper

Surprising findings

• Most LVLMs underperform text-only LLMs fed with lossy Tesseract OCR output, including Gemini-1.5-Pro and Claude-3 Opus.
• Gemini-1.5-Pro and Claude-3 Opus show sharp performance declines when evidence lies on later pages.

Limitations

• Small benchmark scale (135 documents, 1,082 questions) due to annotation cost.
• Open-source LVLMs require image concatenation, confounding comparisons with proprietary models.
• Quality control relies partly on GPT-4o, introducing possible bias toward its answers.

Experimental setup

Datasets: DUDE SlideVQA ChartQA FinanceBench

Benchmarks: MMLongBench-Doc

Metrics: accuracy F1 ANLS

Baselines: GPT-4o GPT-4V Gemini-1.5-Pro Claude-3 Opus GPT-4-turbo InternVL-Chat-v1.5 InternLM-XC2-4KHD Mixtral-Instruct-v0.1-8x22B

Builds on: MATHVISTA MMBench DocVQA SlideVQA DUDE

Closest papers by meaning

• Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.89)
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding (CVPR 2026 · similarity 0.89)
• DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.88)
• MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.88)
• SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.88)
• DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding (CVPR 2026 · similarity 0.88)