Research map / Vision-language and multimodal / Multimodal LLMs
DocVLM: Make Your VLM an Efficient Reader
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DocVLM, which adds a frozen-VLM-compatible OCR branch (DocFormerV2 encoder) compressing OCR words and layout into 64 learned queries fed to the LLM alongside visual tokens, improving low-token document understanding and multipage DocVQA.
Problem
VLMs struggle with document understanding under limited visual token budgets, since high resolution is costly and OCR-as-text prompts are long and underperform full-resolution OCR-free methods.
Approach
Encode OCR text, layout and instruction with an OCR encoder (DocFormerV2 encoder without visual branch) together with M=64 learnable queries, keep only the query outputs, project them, and feed them to a frozen VLM alongside visual tokens, trained in two stages (OCR-LLM alignment, then vision alignment).
Key results
- With 64 learned queries at 448x448 budget, DocVQA improves from 56.0% to 86.6% for InternVL2 (320 total tokens).
- DocVLM with Qwen2-VL at 576 tokens reaches 92.8% DocVQA and 84.5% MP-DocVQA; on multipage with 1024 tokens per page, DocVLM reaches 86.3% ANLS vs. 80.3% for GRAM.
- In LLaVA-OneVision, DocVLM reaches 88.4% DocVQA at 1.5K tokens versus 66.5% for the baseline, and zero-shot DUDE 43.8 vs. 28.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DocVLM with 64 OCR queries | InternVL2 · DocVQA test | ANLS: 56.0 -> 86.6 | Helped |
| Add DocVLM with 64 OCR queries | Qwen2-VL · DocVQA test | ANLS: 84.4 -> 91.2 | Helped |
| Add DocVLM with 64 OCR queries | LLaVA-OneVision · DocVQA test | ANLS: 66.5 -> 88.4 | Helped |
| Replace raw OCR words with DocVLM OCR encoding (ablation) | LLaVA-OneVision, InternVL2, and Qwen2-VL · DocVQA test | ANLS: 85.8 -> 89.4; 84.4 -> 89.2; 89.1 -> 91.9 | Helped |
| Compress OCR encoding to 64 learned queries instead of 800 tokens (ablation) | DocVLM with LLaVA-OneVision, InternVL2, and Qwen2-VL · DocVQA test | ANLS: 89.4 -> 88.4; 89.2 -> 86.6; 91.9 -> 91.2 | Mixed |
| Add Stage II vision alignment (ablation) | DocVLM with LLaVA-OneVision · DocVQA test | ANLS: +6.2; +2.6; +2.1; +0.7 | Helped |
| Add visual features (ablation) | DocVLM with Qwen2 · DocVQA test | ANLS: +4.7; +2.7 | Helped |
| Add visual features (ablation) | DocVLM with Qwen2 · InfoVQA test | ANLS: +3.4; +2.4 | Helped |
| Use page-wise instead of global OCR encoding (ablation) | DocVLM with Qwen2-VL · MP-DocVQA test, 256 and 512 image tokens/page | ANLS: +0.7 | Helped |
| Use page-wise instead of global OCR encoding (ablation) | DocVLM with Qwen2-VL · MP-DocVQA test, 1024 image tokens/page | ANLS: 86.3; 86.3 | No clear effect |
| Add LoRA fine-tuning to the LLM (ablation) | DocVLM with 64 compressed encodings · DocVQA text-only | ANLS: +0.2 | No clear effect |
| Add LoRA fine-tuning to the LLM (ablation) | DocVLM with 64 compressed encodings · InfoVQA text-only | ANLS: +0 | No clear effect |
Surprising findings
- Compressing OCR into 64 learned queries retains most of the gain of uncompressed 800-token OCR encodings (e.g., 88.4 vs 89.4 on LLaVA-OneVision DocVQA).
- Text-only LoRA fine-tuning of the LLM yielded only minor gains for DocVLM encodings, so the authors kept the LLM frozen.
Limitations
- Depends on an external OCR system; OCR errors are not analyzed.
- Multipage results use only single-page training and DocVQA-derived MP-DocVQA is not fully zero-shot.
Experimental setup
Datasets: DocVQA InfoVQA ChartQA TAT-DQA TextVQA ST-VQA OCR-VQA TextCaps COCO Caption VQA-V2
Benchmarks: DocVQA TextVQA ST-VQA InfoVQA TextCaps MP-DocVQA DUDE
Metrics: ANLS VQAScore CIDEr
Baselines: Qwen2-VL InternVL 2 LLaVA-OneVision GRAM DocOwl2 DocFormer v2 ScreenAI Pali-3
Builds on: DocFormerV2 T5 LLaVA-OneVision InternVL2 Qwen2-VL
Closest papers by meaning
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.89)
- LensVLM: Selective Context Expansion for Compressed Visual Representation of Text (NeurIPS 2026 · similarity 0.88)
- DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.88)
- FastVLM: Efficient Vision Encoding for Vision Language Models (CVPR 2025 · similarity 0.88)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.87)
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.