Research map / Large language models / Instruction and fine-tuning
LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LayoutLLM, which pairs a LayoutLMv3 document encoder with a Vicuna/Llama2 7B LLM and trains it with layout instruction tuning (layout-aware pre-training plus LayoutCoT-based SFT) for zero-shot document understanding.
Problem
LLMs and MLLMs used for document understanding under-utilize document layout information, both when layout is given as text coordinates and in existing pre-training and SFT for document MLLMs, limiting zero-shot accuracy and interpretability.
Approach
Integrate a document pre-trained model (LayoutLMv3) as the encoder into an LLM, train it with three levels of layout-aware instruction-tuning pre-training tasks (document, region, segment), and add a LayoutCoT reasoning procedure (question analysis, relevant area localization, answer formation) during SFT.
Key results
- LayoutLLM-7B (Vicuna-1.5-7B backbone) reaches 74.27 ANLS on DocVQA zero-shot, versus 66.99 for plain-text Vicuna-1.5-7B and 65.1 for Qwen-VL-7B (which was trained on DocVQA).
- On zero-shot VIE, LayoutLLM-7B achieves 79.98 on FUNSD, 63.10 on CORD and 72.12 on SROIE, the best among the open 7B LLM/MLLM baselines.
- Ablation on DocVQA/FUNSD: layout-aware pre-training gives +1.49/+3.06 and layout-aware SFT (LayoutCoT) gives +1.96/+5.96 over the SFT-only baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add layout-aware pre-training (ablation) | LayoutLLM · DocVQA test | ANLS: +1.49% | Helped |
| Add layout-aware pre-training (ablation) | LayoutLLM · FUNSD test | ANLS: +3.06% | Helped |
| Add LayoutCoT in layout-aware SFT (ablation) | LayoutLLM with layout-aware pre-training · DocVQA test | ANLS: 1.96% | Helped |
| Add LayoutCoT in layout-aware SFT (ablation) | LayoutLLM with layout-aware pre-training · FUNSD test | ANLS: 5.96% | Helped |
| Use LayoutLLM with layout instruction tuning | Open-source LLMs and MLLMs for document understanding · DocVQA test | ANLS | Helped |
| Use LayoutLLM with layout instruction tuning | Open-source LLMs and MLLMs for document understanding · VisualMRC test | Rouge-L | Helped |
| Use LayoutLLM with layout instruction tuning | Open-source LLMs and MLLMs for document understanding · FUNSD test | ANLS | Helped |
| Use LayoutLLM with layout instruction tuning | Open-source LLMs and MLLMs for document understanding · CORD test | ANLS | Helped |
| Use LayoutLLM with layout instruction tuning | Open-source LLMs and MLLMs for document understanding · SROIE test | ANLS | Helped |
Surprising findings
- Adding explicit layout coordinates as text (Layout Text) did not reliably improve LLMs; for Vicuna-1.5 it improved FUNSD but reduced DocVQA (66.99 to 56.81).
- The zero-shot LayoutLLM outperforms mPLUG-DocOwl and Qwen-VL on DocVQA, even though those models were trained on DocVQA training data.
Limitations
- Cannot refuse false-positive outputs or indicate that the answer is absent from the document.
- Struggles with precise region-level relationship understanding.
- LayoutCoT training data depends on GPT-3.5 Turbo generation and rule-based conversion.
Experimental setup
Datasets: PubLayNet DocLayNet DocBank RVL-CDIP DocILE PubTabNet FeTaQA DocVQA VisualMRC FUNSD CORD SROIE
Benchmarks: DocVQA VisualMRC FUNSD CORD SROIE
Metrics: ANLS Rouge-L F1
Baselines: LayoutLMv3 Llama2-7B Llama2-7B-chat Vicuna-7B Vicuna-1.5-7B LLaVAR-7B LLaVA-1.5-7B mPLUG-DocOwl-7B Qwen-VL-7B
Builds on: LayoutLMv3 LLaVA chain-of-thought prompting Vicuna
Closest papers by meaning
- A Simple yet Effective Layout Token in Large Language Models for Document Understanding (CVPR 2025 · similarity 0.92)
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.89)
- ProcTag: Process Tagging for Assessing the Efficacy of Document Instruction Data (AAAI 2025 · similarity 0.87)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.87)
- Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement (CVPR 2026 · similarity 0.86)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.