Research map / Vision-language and multimodal / CLIP and zero-shot
InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InstructDoc, a 30-dataset, 12-task instruction-tuning collection for visual document understanding, and InstructDr, a BLIP-2/FlanT5-based model with a trainable Document-former bridging document image, OCR+layout, and LLM for zero-shot VDU.
Problem
Visual document understanding tasks (QA, information extraction, classification, etc.) on diverse real-world documents need models that generalize zero-shot to unseen datasets, tasks, and document domains when given natural-language instructions; existing visual instruction-tuned MLLMs struggle with
Approach
Unify 30 VDU datasets under human-written instructions with intent and answer-style annotations, and bridge a CLIP vision encoder and FlanT5 LLM with a Document-former that jointly encodes image features, spatially aware OCR tokens (text plus bounding boxes), and instructions, training only the Document-former and projection layer.
Key results
- InstructDr reaches 43.0 held-out average zero-shot score versus 35.7 for BLIP-2 and 12.9 for InstructBLIP, and outperforms all listed MLLMs on every held-out dataset.
- InstructDr (43.0 held-out avg.) outperforms ChatGPT (32.2) on all held-out datasets, while GPT-4 averages 50.5.
- After task-specific fine-tuning of the Document-former, InstructDr reaches 46.8 ANLS on DUDE and 37.7 EM on SlideVQA, above the supervised SOTA numbers reported (46.1 and 33.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InstructDr instead of GPT-4 | zero-shot VDU evaluation · ChartQA and SlideVQA | Relaxed Accuracy / EM: 20.9 -> 29.4; 21.0 -> 31.9 | Helped |
| Use InstructDr instead of GPT-4 | zero-shot VDU evaluation · FUNSD, CORD, InfoVQA, TabFact, DUDE | eF1 / ANLS / Accuracy: 47.5 -> 38.2; 69.4 -> 46.0; 49.9 -> 50.9; 68.8 -> 59.4; 46.3 -> 45.2 | Hurt |
| Add Document-former (ablation) | InstructDr · CORD | eF1: 38.5 -> 46.0 | Helped |
| Add Document-former (ablation) | InstructDr · TabFact | Accuracy: 58.8 -> 59.4 | Helped |
| Add Document-former (ablation) | InstructDr · DUDE | ANLS: 44.6 -> 45.2 | Helped |
| Add spatially aware OCR features (ablation) | InstructDr · CORD | eF1: 33.8 -> 46.0 | Helped |
| Add spatially aware OCR features (ablation) | InstructDr · DUDE | ANLS: 43.9 -> 45.2 | Helped |
| Use mean pooling instead of concatenation for page features (ablation) | InstructDr · DUDE | ANLS: 43.8 -> 45.2 | Helped |
| Add instructions at test time (ablation) | InstructDr · TabFact | Accuracy: 4.0 -> 59.4 | Helped |
| Add instructions at test time (ablation) | InstructDr · DUDE | ANLS: 38.9 -> 45.2 | Helped |
| Add instructions during training (ablation) | InstructDr · CORD | eF1: 17.3 -> 46.0 | Helped |
| Add instructions during training (ablation) | InstructDr · DUDE | ANLS: 34.0 -> 45.2 | Helped |
Surprising findings
- InstructBLIP, instruction-tuned on scene images, performs worse than its BLIP-2 backbone on the document benchmarks.
- Removing instructions at both train and test time drops the held-out average to 21.3, with CORD eF1 falling to 0.4.
Limitations
- Performance depends heavily on noisy OCR output quality.
- Each instance has a single document-text pair, so the model lacks multi-document in-context learning ability.
- The number of tasks and instruction templates is still limited.
Experimental setup
Datasets: DocILE KLC Deepform PWC Wildreceipt SROIE VisualMRC WebSRC OCRVQA DocVQA HW-SQuAD TAT-DQA
Benchmarks: FUNSD CORD ChartQA InfoVQA TabFact DUDE SlideVQA VisualMRC TextVQA ST-VQA
Metrics: entity F1 F1 relaxed accuracy ANLS accuracy exact match ROUGE-L
Baselines: LLaVA MiniGPT-4 mPLUG-Owl BLIP-2 InstructBLIP LLMDoc LLaVAR ChatGPT GPT-4
Builds on: BLIP-2 InstructBLIP FlanT5 CLIP Q-former
Closest papers by meaning
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.89)
- DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.88)
- Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models (CVPR 2024 · similarity 0.87)
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.87)
- DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming (AAAI 2025 · similarity 0.86)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.