Research map / Vision-language and multimodal / Multimodal LLMs

BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BigDocs-7.5M, a license-permissive 7.5M-sample document/code multimodal training dataset, plus the BigDocs-Bench suite of 10 structured-output tasks, and shows models trained on it beat DocStruct4M and approach or beat GPT-4o on several tasks.

Problem

Open multimodal document-understanding models lack permissively licensed training data, and existing open datasets focus on basic OCR/QA rather than long structured outputs such as HTML, LaTeX, SVG, and JSON from images.

Approach

Curate 133 candidate datasets down to 16 permissively licensed sources plus newly crawled/rendered data, track per-sample licensing and transformation metadata, and pair it with a 10-task benchmark suite (BigDocs-Bench) for image-to-code and GUI reasoning tasks.

Key results

• Phi3.5-Vision-4B + BigDocs reaches 56.34 average on BigDocs-Bench, versus 36.84 for GPT-4o, and tops 8 of 10 tasks.
• On Table2LaTeX human evaluation, Phi3.5 BigDocs wins 88% of comparisons against Phi3.5 Instruct and 63% against GPT-4o.
• On general document benchmarks, Phi3.5-Vision-4B trained on BigDocs averages 60.80 versus 60.51 for DocStruct4M training.

Measured effects reported in the paper

Surprising findings

• Instruction-tuned Qwen2-VL is outperformed on several tasks (TabFact, DeepForm, KLC, TableVQA) by the BigDocs-trained version, which the authors attribute to prompt engineering and task-specific optimization in the instruct model.
• Phi3.5 BigDocs wins only 36% against GPT-4o on Screenshot2HTML in human evaluation, showing the gap is task-dependent.

Limitations

• Suboptimal performance on some public benchmarks, suggesting the data mixture needs refinement.
• Maximum context of 8192 tokens restricts long structured outputs such as HTML and SVG.
• Commercial viability of base models is uncertain because their pretraining data lacks transparency.

Experimental setup

Datasets: BigDocs-7.5M DocStruct4M DocDownStream TabFact WikiTQ TextVQA CORD PubTables-1M SVG-Stack FineWeb SeeClick CDIP-1M

Benchmarks: DocVQA InfoVQA DeepForm KLC WTQ TabFact ChartQA TextVQA MMMU DUDE SlideVQA TableVQA

Metrics: accuracy Tree Edit Distance (DOM) TeXBLEU DINOScore LST F1 RMSF1 Rouge-L F1 human preference win rate

Baselines: DocStruct4M GPT-4o Claude-3.5 Sonnet GeminiPro-1.5 Qwen2-VL-72B Idefics2-8B Llama-3.2.90B DocOwl1.5-8B Qwen2-VL-2B Phi3.5-Vision-4B LLaVA-NeXT-7B

Builds on: DocStruct4M DocOwl1.5 LLaVA-NeXT Qwen2-VL Phi-3.5-Vision

Closest papers by meaning

• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.88)
• SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.88)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
• Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.87)
• VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.87)
• Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.86)