Research map / Vision-language and multimodal / Human and embodied modeling
DocFormerv2: Local Features for Document Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DocFormerv2, an encoder-decoder multimodal transformer for visual document understanding, pre-trained on 64M IDL documents with two encoder tasks (token-to-line, token-to-grid) and one decoder denoising LM task, and fine-tuned for document, table, form, and scene-text VQA.
Problem
Visual document understanding requires jointly modeling text, visual layout, and spatial features; prior encoder-only or heavily CNN-based models lack local layout-relative alignment and are complex to build.
Approach
An encoder-decoder transformer with a simple linear visual embedding (no object detector), trained with asymmetric pre-training: two local-alignment tasks on the encoder (predicting line distance between token pairs and the grid cell of each token) plus masked language modeling with masked spatial features on the decoder.
Key results
- DocFormerv2-large reaches 83.2% on TabFact (+4.3% over UDOP/LayoutLMv3-style prior art) and 88.89% F1 on FUNSD.
- DocFormerv2-large reaches 71.5% test accuracy on OCR-VQA, +3.4% over LaTr-base, and 64.0% on TextVQA test, outperforming Flamingo (80B) by 9.9 points.
- Replacing the linear visual layer with Swinv2 in the small variant lowers DocVQA ANLS from 70.5 to 66.2, supporting the simpler visual branch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Token-to-Line pre-training to visual features (ablation) | DocFormerv2-small · DocVQA | ANLS: 69 -> 71.2 | Helped |
| add Token-to-Grid pre-training to visual features (ablation) | DocFormerv2-small · DocVQA | ANLS: 69 -> 71.7 | Helped |
| add Token-to-Line and Token-to-Grid pre-training to visual features (ablation) | DocFormerv2-small · DocVQA | ANLS: 69 -> 73.0 | Helped |
| add visual features (ablation) | DocFormerv2-small · FUNSD | F1: 82.7 -> 83.3 | Helped |
| add Token-to-Line pre-training to visual features (ablation) | DocFormerv2-small · FUNSD | F1: 83.3 -> 82.5 | Hurt |
| add visual features (ablation) | DocFormerv2-small · ST-VQA | evaluation accuracy | Helped |
| add Token-to-Line pre-training to visual features (ablation) | DocFormerv2-small · ST-VQA | evaluation accuracy: 64.9 -> 64.9 | No clear effect |
| add Token-to-Grid pre-training to visual features (ablation) | DocFormerv2-small · TextVQA | evaluation accuracy: 55.8 -> 54.3 | Hurt |
| increase pre-training data from 11M to 64M documents (ablation) | DocFormerv2-base · FUNSD | F1: 86.1 -> 87.9 | Helped |
| increase pre-training data from 11M to 64M documents (ablation) | DocFormerv2-base · CORD | F1: 96.2 -> 96.8 | Helped |
| replace linear image features with Swinv2-small image encoder (ablation) | DocFormerv2-small · DocVQA | ANLS: 70.5 -> 66.2 | Hurt |
| use extra document VQA data before fine-tuning | DocFormerv2-large · DocVQA test | ANLS: 87.05 -> 87.84 | Helped |
Surprising findings
- On TextVQA test, a document-pretrained model with no natural image-text pre-training outperforms much larger models such as Flamingo (80B).
- Under 20% injected OCR noise, the encoder-decoder model drops only 1.68 F1 points versus 9.84 for an encoder-only model.
Limitations
- Pre-training data (IDL, 64M documents) is proprietary-scale and not released, limiting reproducibility.
- No dataset-specific hyperparameter tuning; some comparisons use different OCR-box setups and are not directly comparable (FUNSD/CORD footnotes).
Experimental setup
Datasets: IDL (Industry Documents Library) 64M TabFact WikiTable DocVQA InfographicsVQA FUNSD CORD OCR-VQA TextVQA ST-VQA
Benchmarks: TabFact WikiTable DocVQA InfographicsVQA FUNSD CORD OCR-VQA TextVQA ST-VQA
Metrics: accuracy ANLS F1
Baselines: LayoutLMv3 UDOP TILT LayoutLMv2 LaTr GIT GIT2 PaLi Flamingo Donut Pix2Struct FormNet
Builds on: TILT DocFormer T5 LayoutLMv2
Closest papers by meaning
- DocVLM: Make Your VLM an Efficient Reader (CVPR 2025 · similarity 0.88)
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.88)
- InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · similarity 0.88)
- Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models (CVPR 2024 · similarity 0.88)
- DAVE: A VLM Vision Encoder for Document Understanding and Web Agents (ICLR 2026 · similarity 0.88)
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.