Research map / Vision-language and multimodal / Human and embodied modeling
Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DoCo, a document-object contrastive pre-training method that aligns the vision encoder of LVLMs with LayoutLMv3 multimodal features of OCR boxes, improving visual document understanding without inference overhead.
Problem
LVLMs underperform on text-rich visual document understanding because image-level CLIP-style contrastive pretraining yields coarse features that miss fine-grained document details (termed fine-grained feature collapse).
Approach
Align patch-level visual features, pooled per OCR bounding box via an ROI Aggregation attention mask, with fine-grained visual-layout-text features from an auxiliary LayoutLMv3 encoder using intra-image and inter-image InfoNCE contrastive losses; the multimodal branch is discarded after pre-training.
Key results
- DoCo improves Qwen-VL-Chat's mean score over the CLIP-pretrained version by about 2% across VDU benchmarks.
- On DocVQA, DoCo lifts Qwen-VL-Chat from 62.2 (w/o DoCo) to 64.8 (+2.6), with Intra-DoCo giving +1.7 and Inter-DoCo +0.9.
- DoCo improves mPLUG-Owl by about 1.8% mean over the CLIP-pretrained version.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DoCo pre-training | Qwen-VL-Chat · OCRVQA | accuracy: 71.1 -> 73.2 | Helped |
| Add DoCo pre-training | Qwen-VL-Chat · TextVQA | accuracy: 61.7 -> 63.6 | Helped |
| Add DoCo pre-training | Qwen-VL-Chat · DocVQA | ANLS: 62.2 -> 64.8 | Helped |
| Add DoCo pre-training | Qwen-VL-Chat · InfoVQA | ANLS: 33.1 -> 34.9 | Helped |
| Add DoCo pre-training | Qwen-VL-Chat · ChartQA | relaxed accuracy: 67.3 -> 68.9 | Helped |
| Add DoCo pre-training | Qwen-VL-Chat · KLC | F1-score: 31.5 -> 33.8 | Helped |
| Add DoCo pre-training | Qwen-VL-Chat · WTQ | accuracy: 24.8 -> 26.9 | Helped |
| Add DoCo pre-training | Qwen-VL-Chat · TextCaps | CIDEr: 112.3 -> 114.5 | Helped |
| Add Intra-DoCo with ROI Aggregation and image, box, and text modalities (ablation) | Qwen-VL-Chat · DocVQA | ANLS: 62.2 -> 63.9 | Helped |
| Add Inter-DoCo (ablation) | Qwen-VL-Chat with Intra-DoCo · DocVQA | ANLS: 63.9 -> 64.8 | Helped |
| Replace ROI Aggregation with Average Aggregation (ablation) | Qwen-VL-Chat with DoCo · DocVQA | ANLS: 64.8 -> 64.2 | Hurt |
| Add box and text modalities to text-only DoCo (ablation) | Qwen-VL-Chat with DoCo · DocVQA | ANLS: 63.7 -> 64.4 | Helped |
Limitations
- Pre-training relies on OCR outputs (PaddleOCR), so quality depends on the OCR engine.
- Ablations are reported only on DocVQA with Qwen-VL-Chat.
- Gains over baselines are modest (around 1-3 points).
Experimental setup
Datasets: CC3M LAION TextVQA DocVQA ChartQA OCRVQA InfoVQA KLC WTQ TextCaps
Benchmarks: OCRVQA TextVQA DocVQA InfoVQA ChartQA KLC WTQ TextCaps
Metrics: accuracy ANLS F1-score relaxed accuracy CIDEr
Baselines: Qwen-VL-Chat with CLIP pre-training mPLUG-Owl with CLIP pre-training mPLUG-DocOwl UReader LLaVAR MiniGPT-4 Qwen-VL
Builds on: CLIP LayoutLMv3 Qwen-VL mPLUG-Owl LLaVA
Closest papers by meaning
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.89)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.88)
- DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.88)
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.87)
- InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · similarity 0.87)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.