Research map / Vision-language and multimodal / Multimodal LLMs
HRVDA: High-Resolution Visual Document Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HRVDA, an OCR-free MLLM that processes 1536x1536 document images by pruning content-agnostic tokens (Swin encoder with PSENet content detector) and instruction-agnostic tokens (instruction filtering module) before a LLaMA-2-7B LLM, trained on a document instruction-tuning dataset.
Problem
MLLMs underperform on visual document understanding because low-resolution inputs blur text and high-resolution inputs create too many visual tokens for LLM context and compute budgets.
Approach
Filter visual tokens twice: a pluggable PSENet content detector gates Swin Transformer blocks so blank-background windows skip attention, and a fusion Transformer plus 2-layer MLP drops tokens irrelevant to the instruction before the LLM, enabling 1536-resolution inputs at near low-resolution cost.
Key results
- HRVDA reaches 89.3 on CORD and 91.0 on SROIE (entity F1), above the best prior OCR-free baselines in the table (SeRum 84.9 and 85.8).
- On DeepForm, HRVDA scores 63.2 versus 50.7 for SeRum and 49.5 for UReader, and on ChartQA it scores 67.6 versus 66.3 for Qwen-VL.
- With both filtering thresholds at 0.5, forward inference runs about 61% faster than Qwen-VL per the paper's latency table (total 3.64 vs 9.47).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HRVDA instead of SeRum | OCR-free document understanding · CORD | F1: 84.9 -> 89.3 | Helped |
| Use HRVDA instead of SeRum | OCR-free document understanding · SROIE | F1: 85.8 -> 91.0 | Helped |
| Use HRVDA instead of Pix2Struct | OCR-free document understanding · DocVQA | ANLS: 76.6 -> 72.1 | Hurt |
| Use HRVDA instead of UReader | OCR-free document understanding · InfoVQA | ANLS: 42.2 -> 43.5 | Helped |
| Use HRVDA instead of Donut | OCR-free document understanding · DeepForm | F1: 61.6 -> 63.2 | Helped |
| Use HRVDA instead of UReader | OCR-free document understanding · Kleister Charity | F1: 32.8 -> 37.5 | Helped |
| Use HRVDA instead of UReader | OCR-free document understanding · WTQ | accuracy: 29.4 -> 31.2 | Helped |
| Use HRVDA instead of UReader | OCR-free document understanding · TabFact | accuracy: 67.6 -> 72.3 | Helped |
| Use HRVDA instead of UReader | OCR-free document understanding · ChartQA | relaxed accuracy: 59.3 -> 67.6 | Helped |
| Use HRVDA instead of CogVLM | OCR-free document understanding · TextVQA | accuracy: 69.7 -> 73.3 | Helped |
| Use HRVDA instead of UReader | OCR-free document understanding · VisualMRC | CIDEr: 221.7 -> 211.5 | Hurt |
| Use HRVDA instead of CogVLM | OCR-free document understanding · TextCaps | CIDEr: 144.9 -> 125.3 | Hurt |
Surprising findings
- Accuracy peaks at a 50% filtering threshold and declines beyond it, suggesting pruning tokens can improve performance.
- Query-based (Q-Former-style) compression needed about 500 queries at high resolution to avoid degradation, offering no speed advantage.
Limitations
- Struggles with extremely text-dense images and complex structural relations.
- Extreme-aspect-ratio images require multiple crops, which compromises global structure.
- Fixed 1536 resolution limited by GPU memory; no further resolution increase explored.
Experimental setup
Datasets: IIT-CDIP CORD SROIE DocVQA InfographicsVQA DeepForm Kleister Charity WikiTableQuestions TabFact ChartQA TextVQA TextCaps
Benchmarks: CORD SROIE DocVQA InfoVQA DeepForm Kleister Charity WikiTableQuestions TabFact ChartQA TextVQA VisualMRC TextCaps
Metrics: entity-level F1 ANLS accuracy CIDEr relaxed accuracy latency
Baselines: Donut SeRum Pix2Struct Qwen-VL mPLUG-DocOwl UReader CogVLM
Builds on: Swin Transformer LLaMA-2 Donut BLIP-2 PSENet
Closest papers by meaning
- Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models (CVPR 2024 · similarity 0.89)
- DocVLM: Make Your VLM an Efficient Reader (CVPR 2025 · similarity 0.89)
- InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · similarity 0.89)
- Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding (NeurIPS 2024 · similarity 0.88)
- DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.88)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.