Research map / Vision-language and multimodal / Human and embodied modeling
Unifying Vision, Text, and Layout for Universal Document Processing
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the proposed self-supervised pretraining objectives instead of MLM alone (ablation) | UDOP · DocVQA validation | score: 79.7 $\pm$ 0.4 -> 84.4 $\pm$ 0.2 | Improved |
| Add the proposed self-supervised pretraining objectives instead of MLM alone (ablation) | UDOP · RVL-CDIP validation | score: 95.3 $\pm$ 0.3 -> 96.2 $\pm$ 0.2 | Improved |
| Add Visual Text Recognition to Joint Text-Layout pretraining (ablation) | UDOP · DocVQA validation | score: 82.8 $\pm$ 0.1 -> 83.3 $\pm$ 0.2 | Improved |
| Add Layout Modeling after Visual Text Recognition (ablation) | UDOP · DocVQA validation | score: 83.3 $\pm$ 0.2 -> 84.0 $\pm$ 0.3 | Improved |
| Add Layout Modeling after Visual Text Recognition (ablation) | UDOP · RVL-CDIP validation | score: 95.4 $\pm$ 0.2 -> 95.6 $\pm$ 0.2 | Improved |
| Add Masked Image Reconstruction to text-layout objectives (ablation) | UDOP · RVL-CDIP validation | score: 95.6 $\pm$ 0.2 -> 96.2 $\pm$ 0.2 | Improved |
| Add supervised pretraining data after self-supervised objectives (ablation) | UDOP · DocVQA validation | score: 84.4 $\pm$ 0.2 -> 85.0$\pm$ 0.2 | Improved |
| Add supervised pretraining data after self-supervised objectives (ablation) | UDOP · RVL-CDIP validation | score: 96.2 $\pm$ 0.2 -> 96.3 $\pm$ 0.1 | Improved |
| Use a unified encoder instead of separate modality-specific encoders (ablation) | UDOP · DUE Benchmark | average score: 64.7$\pm$0.3 -> 65.1$\pm$0.2 | Improved |
| Use a unified encoder instead of separate modality-specific encoders (ablation) | UDOP · WTQ | score: 46.7$\pm$0.4 -> 47.2$\pm$0.2 | Hurt |
| Remove image input embeddings (ablation) | UDOP · DocVQA | score: 84.7 -> 84.4 | Hurt |
| Remove image input embeddings (ablation) | UDOP · InfoVQA | score: 47.4 -> 45.0 | Hurt |
Closest papers by meaning
- A Simple yet Effective Layout Token in Large Language Models for Document Understanding (CVPR 2025 · similarity 0.83)
- Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training (CVPR 2026 · similarity 0.82)
- Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation (NeurIPS 2026 · similarity 0.82)
- Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding (CVPR 2025 · similarity 0.82)
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.82)
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
