Research map / Large language models / AI and society
MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MDPBench, a 3,400-image benchmark for multilingual digital and photographed document parsing across 17 languages, and evaluates general VLMs, specialized VLMs, and pipeline tools.
Problem
Document parsing benchmarks focus on clean digital pages in a few high-resource languages, leaving multilingual and photographed (real-world captured) document parsing unevaluated.
Approach
A multilingual benchmark pairing digital-born documents with printed/screen-recaptured photographs under varied conditions, annotated via expert-model consensus, manual correction, and human verification, with page-level metric aggregation and a public/private split.
Key results
- Gemini-3-pro-preview achieves 86.4% overall accuracy, the best result, with 89.8% on private split.
- Best open-source model dots.mocr reaches 80.5% overall, with photographed documents averaging a 17.8% drop across models.
- Non-Latin-script languages show an average 14.0% drop relative to Latin-script languages.
Surprising findings
- Several open-source specialized models, such as MinerU-2.5-VLM and MonkeyOCR-pro-3B, score below 10% on Arabic and Hindi despite strong Latin-script performance.
- Models trained mainly on English and Chinese generalize to Latin-script languages like German but collapse on non-Latin scripts.
Limitations
- Private split evaluation requires submission to an official website, limiting open access.
- Photographed documents are captured under specific printed or screen conditions, which may not cover all real-world variation.
- Pipeline annotations rely on expert models that may share biases with evaluated models.
Experimental setup
Datasets: OmniDocBench
Benchmarks: MDPBench
Metrics: Normalized Edit Distance (1-NED) CDM TEDS PageIoU
Baselines: Gemini-3-pro-preview kimi-K2.5 Doubao-2.0-pro Claude-Sonnet-4.6 ChatGPT-5.2 Qwen3-VL-8B Qwen3.5-9B InternVL-3.5-8B dots.mocr PaddleOCR-VL-1.5 dots.ocr olmOCR2
Builds on: OmniDocBench DocPTBench Real5-OmniDocBench
Closest papers by meaning
- DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation (NeurIPS 2026 · similarity 0.93)
- MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing (NeurIPS 2026 · similarity 0.90)
- Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training (CVPR 2026 · similarity 0.88)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.87)
- OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations (CVPR 2025 · similarity 0.87)
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.