Research map / Large language models / Language, safety and interpretability

DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Text generation and evaluation metrics · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DocPTBench, a benchmark of ~1,381 photographed and unwarped document images (derived from OmniDocBench) for end-to-end parsing and translation, showing large performance drops for MLLMs and expert parsers versus digital-born originals.

Problem

Existing document parsing and translation benchmarks use pristine digital-born or scanned documents, so robustness of end-to-end MLLM and expert parsers to real-world photographed document conditions (geometric and photometric distortions) is unverified.

Approach

A three-tier benchmark (Original, Photographed, Unwarping) built on OmniDocBench pages, with photographic simulations and physically captured photos, paired with human-verified parsing annotations and eight-direction translation references, evaluated with a unified dual-task protocol comparing Simple vs. Chain-of-Thought prompting.

Key results

• Photographed documents raise MLLM parsing edit distance by about 18% on average and expert parsers by about 25%, per the paper's summary.
• Translation BLEU drops about 12% on average when moving from original to photographed images; e.g., Qwen-VL-Max En-Zh CoT BLEU falls from 47.60 to 37.44.
• Unwarping recovers much of the loss; e.g., DeepSeek-OCR En overall Edit improves from 54.4 (photographed) to 22.1 (unwarped) vs 13.4 original.

Surprising findings

• Some smaller open-source models fail to follow the translation instruction and revert to plain OCR, which Chain-of-Thought prompting partially corrects.
• Doubao-Seed-1.6-vision parses well but translates only moderately, attributed to a weaker text-only translation engine.

Limitations

• Photographed set is mostly simulated (981 of 1,381 images); only 100 documents physically photographed.
• Translation references were produced with Qwen-Max and then manually verified, which may bias toward that model's style.
• Unwarping relies on a commercial API whose effect is not separately characterized.

Experimental setup

Datasets: OmniDocBench

Benchmarks: DocPTBench

Metrics: Edit distance TEDS BLEU chrF METEOR STEDS

Baselines: PaddleOCR-VL MinerU2.5 dots.ocr MonkeyOCR DeepSeek-OCR olmOCR2 Gemini2.5-Pro Doubao-Seed-1.6-vision Qwen-VL-Max GLM-4.5v Kimi-VL Qwen3-VL-4B

Builds on: OmniDocBench DITrans DoTA Chain-of-Thought prompting

Closest papers by meaning

• MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios (NeurIPS 2026 · similarity 0.93)
• Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training (CVPR 2026 · similarity 0.90)
• MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing (NeurIPS 2026 · similarity 0.87)
• OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations (CVPR 2025 · similarity 0.86)
• SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.85)
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.85)