Research map / Vision-language and multimodal / Multimodal LLMs

MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DocWeaver, an LLM-based multi-agent pipeline that auto-generates MosaicDoc, a large bilingual (Chinese/English) newspaper and magazine benchmark for visually rich document understanding, and evaluates 13 VLMs on it.

Problem

Existing document understanding benchmarks are English-centric, use visually simple layouts, and cover few tasks, so they do not evaluate VLMs on visually rich documents such as newspapers and magazines with complex, non-Manhattan layouts.

Approach

A fully automated multi-agent pipeline (layout detection, multi-engine OCR, HTML/PDF alignment for reading order, LLM question generation with multi-span emphasis, and LLM-based hallucination guardrails) generates multi-task annotations for complex bilingual newspapers and magazines at scale.

Key results

• General VLMs such as Qwen2.5-VL and Gemini-2.5 outperform expert models on DocVQA, but all models show a sharp ANLSL drop on multi-span questions.
• Gemini-2.5 achieves 87.9 line-level reading-order F1 on English magazines and 87.9/56.3 on English/Chinese newspapers, while most models show high precision but low recall.
• Page-level OCR is much worse on newspapers than magazines; for example, Gemini-2.5 CRR drops from 89.42 (en magazine) to 87.90 (en newspaper) and 66.64 (zh newspaper).

Surprising findings

• Expert VLMs using token reduction (TextMonkey, mPLUG-DocOwl2) struggle unexpectedly on information-dense magazine and newspaper pages.
• Models frequently fall into repetitive output loops during long-page OCR, which lowers OCRR and CRR.

Limitations

• Human validation covers only a 200-image test sample; the bulk of the benchmark relies on automated LLM generation and filtering.
• Reliance on GPT-4o and DeepSeek-R1 generators and QwQ-32B judge may introduce model-specific biases.
• Spatial grounding is near-zero on complex layouts for most models, so the grounding task is only partially covered.

Experimental setup

Datasets: M6Doc MultiSpanQA InfographicsVQA TableBench MP-DocVQA TAT-DQA DuReader-vis DUDE ROOR ReadingBank olmOCR-mix-0225

Benchmarks: MosaicDoc

Metrics: ANLSL CRR OCRR Micro-F1 (reading order) BLEU precision/recall/F1 for grounding

Baselines: LayoutReader Donut ViTLP Vary TextMonkey mPLUG-DocOwl2 GOT-OCR olmOCR CogVLM2 InternVL3 Qwen2.5-VL GPT-4o

Builds on: G-EVAL ROOR TVG MinerU PP-DocLayout

Closest papers by meaning

• MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.88)
• DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.88)
• Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.87)
• InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · similarity 0.87)
• DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.87)
• DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming (AAAI 2025 · similarity 0.87)