Research map / Vision-language and multimodal / Multimodal LLMs
DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DocLayLLM, an OCR-dependent multimodal extension of LLMs that feeds patch tokens and layout embeddings directly into the LLM, trained with CoT pre-training and CoT annealing using LoRA.
Problem
Efficient text-rich document understanding (TDU) that integrates OCR text, layout and visual information into LLMs without heavy computation or an extra document encoder.
Approach
Integrate visual patch tokens and 2D layout embeddings (via lightweight embedders initialized from LayoutLMv3) directly into an LLM's input so the LLM itself encodes the OCR text; generate CoT-style QA data for pre-training and anneal SFT data from CoT to direct-answer format.
Key results
- DocLayLLM Llama3 reaches 68.46 average on DocVQA/VisualMRC and 79.94 KIE average in the LayoutLLM-metric comparison, beating LayoutLLM and LayTextLLM.
- Zero-shot DocLayLLM Llama3 gets 77.79 DocVQA and 80.28 FUNSD, outperforming DocLLM on most datasets without downstream training.
- With downstream training data, DocLayLLM Llama3 averages 157.60 across DocVQA/InfoVQA/VisualMRC and 70.99 on table understanding, surpassing OCR-free methods such as DocOwl 1.5 and TextSquare in most datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CoT pre-training (ablation) | DocLayLLM Llama3 · DocVQA | score: 76.00 -> 77.43 | Helped |
| add CoT pre-training (ablation) | DocLayLLM Llama3 · DeepForm | score: 24.54 -> 35.41 | Helped |
| use CoT annealing instead of SFT with only CoT data (ablation) | DocLayLLM Llama3 · DocVQA | score: 77.40 -> 77.43 | Helped |
| use CoT annealing instead of SFT with only CoT data (ablation) | DocLayLLM Llama3 · VisualMRC | score: 310.17 -> 316.27 | Helped |
| initialize visual and layout embedders with pretrained weights instead of random initialization (ablation) | DocLayLLM Llama3 · DeepForm | score: 31.31 -> 35.41 | Helped |
| add embedded layout coordinates to text-only input (ablation) | DocLayLLM · DeepForm | score: 15.76 -> 33.75 | Helped |
| add OCR coordinates as text to text-only input (ablation) | DocLayLLM · DocVQA | score: 73.57 -> 75.72 | Helped |
| add visual patch features to text and embedded layout (ablation) | DocLayLLM · DeepForm | score: 33.75 -> 35.41 | Helped |
| encode OCR coordinates with a layout embedder instead of as text (ablation) | DocLayLLM · DocVQA | average input length: 1571.80 -> 455.17 | Helped |
| use DocLayLLM instead of TextSquare | OCR-free comparison · TabFact | score: 84.20 -> 83.36 | Hurt |
Surprising findings
- Using the commercial OCR engine on DocVQA gives higher ANLS (87.52) than the official OCR (86.52), suggesting reported results underestimate the model.
- Encoding layout as text coordinates inflates the OCR input length roughly 3-5x compared with the layout embedder (e.g., 4952.87 vs 1198.97 tokens on DeepForm).
Limitations
- Requires an external OCR engine, so performance depends on OCR quality.
- Fixed 224x224 image resolution (196 patch tokens) limits fine-grained text perception in dense documents.
Experimental setup
Datasets: RVL-CDIP PubLayNet PubTabNet DocLayNet DocBank DocILE LayoutLLM document dense description data LayoutLLM layout-aware document QA data FUNSD CORD SROIE POIE
Benchmarks: DocVQA VisualMRC FUNSD CORD SROIE POIE InfoVQA DeepForm KLC WTQ TabFact
Metrics: ANLS F1 accuracy precision
Baselines: LayoutLLM LayTextLLM DocLLM LMDX ICL-D3IE DocOwl 1.5 TextMonkey TextSquare DocKylin Monkey UReader LLaVAR
Builds on: LayoutLMv3 LayoutLLM LoRA chain-of-thought XY-Cut
Closest papers by meaning
- A Simple yet Effective Layout Token in Large Language Models for Document Understanding (CVPR 2025 · similarity 0.91)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.90)
- LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding (CVPR 2024 · similarity 0.89)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.88)
- URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding (AAAI 2026 · similarity 0.88)
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.