Research map / Vision-language and multimodal / Human and embodied modeling
Layout-aware document understanding
Extracts information from visually rich documents with layout-aware pretrained models.
73 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Human and embodied modeling, keyword cluster “pre · generation · unified”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 1 tracked benchmark, but the best result has barely moved since 2024.
31 papers in 2023–24 and 26 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 5.8% to 3.9%.
What looks settled
- FUNSD (F1): best up to 2024 92.86, best in 2025–26 88.23 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Learning to Extract Semantic Structure From Documents Using Multimodal Fully Convolutional Neural Networks (CVPR 2017 · 255 citations)
- 2019 · Hierarchical Encoder with Auxiliary Supervision for Neural Table-to-Text Generation: Learning Better Representation for Tables (AAAI 2019)
- 2020 · Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism (AAAI 2020 · 30 citations)
- 2021 · SelfDoc: Self-Supervised Document Representation Learning (CVPR 2021 · 122 citations)
- 2022 · BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents (AAAI 2022 · 129 citations)
- 2023 · RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL (AAAI 2023 · 178 citations)
- 2024 · DocFormerv2: Local Features for Document Understanding (AAAI 2024 · 52 citations)
- 2025 · MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures (CVPR 2025 · 5 citations)
- 2026 · MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures (CVPR 2026 · 1 citation)
Earliest papers here
- Learning to Extract Semantic Structure From Documents Using Multimodal Fully Convolutional Neural Networks (CVPR 2017 · 255 citations)
- Hierarchical Encoder with Auxiliary Supervision for Neural Table-to-Text Generation: Learning Better Representation for Tables (AAAI 2019)
- Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism (AAAI 2020 · 30 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
FUNSD · F1 (higher is better)
4 papers, 2023–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 92.86 | GeoLayoutLM: Geometric Pre-Training for Visual Information Extraction (CVPR 2023 · 53 citations) |
What has been tried
58 papers here report 638 measured changes (a component added, removed or swapped, compared with a baseline): helped 73%, no clear effect 5%, hurt 20%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add Token-to-Line pre-training to visual features | DocVQA · ANLS | 69 -> 71.2 | Helped | DocFormerv2: Local Features for Document Understanding (AAAI 2024 · 52 citations) |
| add spatial-window prompting | Total-Text · end-to-end recognition (None) | 82.4 -> 82.9 | Helped | OmniParser: A Unified Framework for Text Spotting Key Information Extraction and Table Recognition (CVPR 2024 · 46 citations) |
| Add Intra-DoCo with ROI Aggregation and image, box, and text modalities | DocVQA · ANLS | 62.2 -> 63.9 | Helped | Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models (CVPR 2024 · 28 citations) |
| Add random token replacement during training | Out-of-domain documents · failed page conversions | 32% decline | Helped | Nougat: Neural Optical Understanding for Academic Documents (ICLR 2024 · 20 citations) |
| Add early fusion | M6Doc · mAP | 68.5 -> 69.9 | Helped | M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis (AAAI 2024 · 17 citations) |
| use full-page context rather than crop-only embeddings for character clustering | PopManga (Test-S) · AMI | 0.4342 -> 0.6551 | Helped | The Manga Whisperer: Automatically Generating Transcriptions for Comics (CVPR 2024 · 14 citations) |
| Add both sentence and EDU guidance rather than EDU guidance alone | CNN/Daily Mail · Entail | 0.916 -> 0.929 | Helped | Improving Faithfulness in Abstractive Text Summarization with EDUs Using BART (Student Abstract) (AAAI 2024 · 7 citations) |
| add the OCSR encoder to the VTL encoder | USPTO-Markush · CXSMILES EM | 23 -> 32 | Helped | MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures (CVPR 2025 · 5 citations) |
| Add MTIM | FUNSD and XFUND zero-shot · F1 | 84.55 -> 86.24; 57.25 -> 57.65 | Helped | LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretraining (AAAI 2025 · 4 citations) |
| Use the proposed fine-grained image-text fusion instead of feature concatenation | DocMSU · Accuracy | 90.27 -> 97.83 | Helped | DocMSU: A Comprehensive Benchmark for Document-Level Multimodal Sarcasm Understanding (AAAI 2024 · 4 citations) |
| add MLM and LAMIM pretraining objectives | PatentDesc-355K test, brief · Avg. BLEU | 40.86 -> 43.39 | Helped | PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figures (AAAI 2025 · 3 citations) |
| Use three-stage progressive training instead of end-to-end training | Training convergence · convergence behavior | Helped | ScreenMark: Watermarking Arbitrary Visual Content on Screen (AAAI 2025 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- RVL-CDIP · 2 of 17 papers
- FUNSD · 2 of 17 papers
- XFUND · 2 of 17 papers
- CORD · 2 of 17 papers
- ImageNet · 2 of 17 papers
Baselines
- Dolphin-v2 · 2 of 17 papers
- DeepSeek-OCR2 · 2 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 57 papers; from OpenAlex and the papers' affiliations).
- Alibaba Group · 5 papers
- Peking University · 4 papers
- University of Science and Technology of China · 4 papers
- Microsoft · 4 papers
- Renmin University of China · 2 papers
- IBM · 2 papers
- Huawei · 2 papers
- Huazhong University of Science and Technology · 2 papers
- Google · 2 papers
- Meta · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Vision-language pretraining (182 papers)
- Medical vision-language models (130 papers)
- Text-driven human motion generation (36 papers)
Elsewhere in the topic
- Scene graph generation (126 papers)
- Language-grounded scene text and structure (124 papers)
- fMRI visual decoding (106 papers)
- Human-like visual cognition in VLMs (97 papers)
- Multimodal and affective interaction modeling (87 papers)
- Multimodal retrieval-augmented modeling (85 papers)
- Language-grounded visual concept learning (83 papers)
- Composed and text-based image retrieval (78 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).