Research map / Vision-language and multimodal / Multimodal LLMs
Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an OCR-free document understanding framework on pretrained MLLMs (BLIP-2 and mPLUG-Owl) that uses multi-scale visual inputs, a Hierarchical Visual Feature Aggregation module with cross-attentive pooling to cut LLM visual tokens, and relative text-position prediction tasks.
Problem
OCR-free document understanding must handle text of varying font sizes and scales, but multi-scale visual inputs greatly increase LLM cost, and reading full document text in training often truncates due to LLM context limits.
Approach
A feature pyramid over adjacent scales where the finer-scale features are max-pooled and refined via cross-attention (pooled features as queries, original features as keys/values), then added to coarser features so the LLM sees the original token count; plus Reading Partial Text and Predicting Text Position tasks with relative-position targets to avoid truncation and learn layout.
Key results
- With mPLUG-Owl-7B, the method reaches 72.7 on DocVQA and 45.9 on InfoVQA, outperforming UReader (65.4 and 42.2) on all 10 benchmarks.
- With BLIP-2-OPT-2.7B, multi-scale HVFA improves DocVQA from 35.17 (no multi-scale) to 51.38 with all components.
- Cross-attentive pooling beats max-pooling-only and linear projectors on DocVQA (51.38 vs 43.23 and 46.11) in the BLIP-2 ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed framework instead of Monkey | OCR-free document understanding · DocVQA test | ANLS: 66.5 -> 72.7 | Helped |
| use the proposed framework instead of UReader | mPLUG-Owl-7B · DeepForm test | F1: 49.5 -> 53.0 | Helped |
| use the proposed framework instead of UReader | mPLUG-Owl-7B · KLC test | F1: 32.8 -> 36.7 | Helped |
| use the proposed framework instead of Qwen-VL | OCR-free document understanding · ChartQA test | relaxed accuracy: 65.7 -> 63.3 | Hurt |
| add multi-scale visual features with HVFA (ablation) | BLIP-2-OPT-2.7B · DocVQA test | ANLS: 35.17 -> 45.22 | Helped |
| add reconstruction loss to MS + HVFA (ablation) | BLIP-2-OPT-2.7B · DocVQA test | ANLS: 45.22 -> 47.36 | Helped |
| add RTPP to MS + HVFA (ablation) | BLIP-2-OPT-2.7B · DocVQA test | ANLS: 45.22 -> 50.46 | Helped |
| use cross-attentive pooling rather than max pooling only (ablation) | HVFA · DocVQA test | ANLS: 43.23 -> 51.38 | Helped |
| initialize query tokens with max-pooled features rather than random vectors (ablation) | HVFA · DocVQA test | ANLS: 48.07 -> 51.38 | Helped |
| remove stop-gradient (ablation) | HVFA reconstruction loss · DocVQA test | ANLS: 51.38 -> 49.65 | Hurt |
| increase reconstruction-loss weight from λ = 0.1 to λ = 1.0 (ablation) | BLIP-2-OPT-2.7B · DocVQA test | ANLS: 51.38 -> 49.90 | Hurt |
| replace RPT with RFT text-reading task (ablation) | BLIP-2-OPT-2.7B · DocVQA test | ANLS: 49.23 -> 50.46 | Hurt |
Surprising findings
- Using two scales with HVFA is reported as more accurate than the three-scale variant trained without HVFA being infeasible on A100s, with throughput dropping from 1.088 to 0.494 img/s when going to three scales.
- Without stop-gradient, the reconstruction MSE loss collapses to zero early in training, degrading performance.
Limitations
- Trained only on English documents, so may struggle with other languages.
- Restricted to resampler-based MLLMs; not tested on resampler-free architectures.
- Still trails OCR-based SoTA such as LayoutT5 on VisualMRC and large foundation models on DocVQA.
Experimental setup
Datasets: DocVQA InfographicsVQA DeepForm KleisterCharity WikiTableQuestions TabFact ChartQA VisualMRC TextVQA TextCaps
Benchmarks: DocVQA InfographicsVQA DeepForm KleisterCharity WikiTableQuestions TabFact ChartQA VisualMRC TextVQA TextCaps
Metrics: ANLS F1 accuracy relaxed accuracy CIDEr
Baselines: UReader Donut Dessurt Pix2Struct Qwen-VL Monkey
Builds on: UReader BLIP-2 mPLUG-Owl shape-adaptive cropping LoRA MViT
Closest papers by meaning
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.88)
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.87)
- DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming (AAAI 2025 · similarity 0.87)
- Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models (CVPR 2024 · similarity 0.86)
- MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios (NeurIPS 2025 · similarity 0.86)
- DocVLM: Make Your VLM an Efficient Reader (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.