Research map / Vision-language and multimodal / Human and embodied modeling
UniDoc: Unified Pretraining Framework for Document Understanding
Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, Tong Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine vision and language instead of using either stream alone (ablation) | UDoc without pretraining · FUNSD | F1: 77.49, 78.46 -> 83.34 | Improved |
| Add VCL to MSM, training the visual backbone end-to-end (ablation) | UDoc pretraining · FUNSD | F1: 84.37 -> 86.87 | Improved |
| Add VCL to MSM, training the visual backbone end-to-end (ablation) | UDoc pretraining · CORD† | F1: 97.44 -> 98.70 | Improved |
| Add VCL to MSM, training the visual backbone end-to-end (ablation) | UDoc pretraining · RVL-CDIP | Accuracy: 93.10 -> 93.59 | Improved |
| Add VLA to MSM + VCL (ablation) | UDoc pretraining · FUNSD | F1: 86.87 -> 87.38 | Improved |
| Add VLA to MSM + VCL (ablation) | UDoc pretraining · CORD† | F1: 98.70 -> 98.75 | Improved |
| Add VLA to MSM + VCL (ablation) | UDoc pretraining · RVL-CDIP | Accuracy: 93.59 -> 93.92 | Improved |
| Add image-text matching (REL) to MSM + VCL + VLA (ablation) | UDoc pretraining · FUNSD | F1: 87.38 -> 87.20 | Hurt |
| Add image-text matching (REL) to MSM + VCL + VLA (ablation) | UDoc pretraining · CORD† | F1: 98.75 -> 98.13 | Hurt |
| Add image-text matching (REL) to MSM + VCL + VLA (ablation) | UDoc pretraining · RVL-CDIP | Accuracy: 93.92 -> 93.64 | Hurt |
| Add masked language modeling to UDoc pretraining (ablation) | UDoc · FUNSD | F1: 87.38 -> 83.76 | Hurt |
| Add masked language modeling to UDoc pretraining (ablation) | UDoc · CORD | F1: 98.75 -> 98.63 | Hurt |
Closest papers by meaning
- DocMamba: Efficient Document Pre-training with State Space Model (AAAI 2025 · similarity 0.86)
- InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · similarity 0.86)
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.84)
- In-Context Pretraining: Language Modeling Beyond Document Boundaries (ICLR 2024 · similarity 0.84)
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.84)
- DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
