Research map / Large language models / Retrieval and knowledge
OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OmniDocBench, a 981-page, 9-type PDF document parsing benchmark with multi-level annotations and an end-to-end/task/attribute evaluation pipeline, and benchmarks pipeline tools and VLMs on it.
Problem
Existing document parsing benchmarks cover narrow document types, rely on generic text-similarity metrics that poorly assess LaTeX/HTML formulas and tables, and give only overall scores without fine-grained insight.
Approach
A diverse, densely annotated page-level benchmark (19 layout categories, 15 attribute labels, reading order, affiliation, LaTeX/HTML content) paired with a matching-based evaluation pipeline that uses format-specific metrics (CDM, TEDS, normalized edit distance) and an ignore logic for inconsistently-handled components.
Key results
- Pipeline tools MinerU and Mathpix lead on overall end-to-end parsing (overall Edit 0.15 EN for MinerU), with general VLMs such as Qwen2-VL and GPT-4o competitive.
- Most VLMs fail on newspapers (e.g., GPT-4o overall text edit 0.751 on Newspaper vs MinerU 0.984 on the same column in Table 3 reversed), while pipeline tools handle dense layouts better.
- DocLayout-YOLO achieves 47.38 average mAP on the layout subset, outperforming other layout detectors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Qwen2-VL-72B instead of the other evaluated parsers | document parsing · OmniDocBench (9 page types) | TextEdit (overall): 0.206 -> 0.179 | Helped |
| use MinerU instead of Nougat | document parsing · OmniDocBench English | OverallEdit: 0.452 -> 0.15 | Helped |
| use MinerU instead of Nougat | document parsing · OmniDocBench Chinese | OverallEdit: 0.973 -> 0.357 | Helped |
| use DocLayout-YOLO instead of the other evaluated layout detectors | layout detection · OmniDocBench layout subset | mAP: 28.84 -> 47.38 | Helped |
| use RapidTable instead of the other evaluated table recognizers | table recognition · OmniDocBench table subset | overall table recognition: 73.6 -> 82.5 | Helped |
| use GPT-4o instead of GOT-OCR | formula recognition · OmniDocBench formula subset | CDM: 74.1 -> 86.8 | Helped |
| use general VLMs rather than pipeline tools on handwritten notes | end-to-end text recognition · Notes | TextEdit | Mixed |
| use VLMs rather than pipeline tools on newspapers | end-to-end text recognition · Newspaper | TextEdit | Hurt |
| use VLMs on fuzzy scans, watermarks, and colorful backgrounds | end-to-end text recognition · Pages with visual degradations | TextEdit | Helped |
| use multi-column or complex layouts rather than simpler layouts | reading order prediction · Pages with different layout types | Normalized Edit Distance | Hurt |
Surprising findings
- Pipeline tools handle newspapers far better than general VLMs, which largely fail on dense newspaper layouts due to input resolution and token limits.
- Nougat scores near 1.0 edit distance on several page types, indicating near-total failure outside academic papers.
Limitations
- Benchmark is small at 981 pages, limiting statistical robustness for some attribute groups.
- Ignore logic excludes headers, footers, and captions from metrics, which may hide real differences between models.
Experimental setup
Datasets: OmniDocBench
Benchmarks: OmniDocBench
Metrics: normalized edit distance TEDS CDM BLEU mAP reading order edit distance
Baselines: MinerU Marker Mathpix GOT-OCR Nougat GPT-4o Qwen2-VL-72B InternVL2-76B
Builds on: Nougat GOT-OCR UniMERNet LayoutLMv3 PaddleOCR
Closest papers by meaning
- MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing (NeurIPS 2026 · similarity 0.89)
- MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios (NeurIPS 2026 · similarity 0.87)
- Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training (CVPR 2026 · similarity 0.86)
- DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation (NeurIPS 2026 · similarity 0.86)
- OmniParser: A Unified Framework for Text Spotting Key Information Extraction and Table Recognition (CVPR 2024 · similarity 0.86)
- ColPali: Efficient Document Retrieval with Vision Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.