Research map / Vision-language and multimodal / Multimodal LLMs
DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DocKylin, a document-focused MLLM that slims visual content via parameter-free pixel-level redundancy removal (APS) and token-level dual-center clustering with similarity-weighted aggregation (DTS) to shorten visual sequences for document understanding.
Problem
High-resolution document images produce long, redundant visual token sequences that raise computational cost and strain MLLMs' long-context handling, while fixed resolutions distort text and fixed compression ratios ignore varying content density.
Approach
Parameter-free Adaptive Pixel Slimming uses Sobel gradients to remove low-information rows/columns before encoding, and Dynamic Token Slimming clusters visual tokens into two groups (essential vs. nonessential, identified by similarity) and merges nonessential tokens into essential ones via similarity-weighted summation.
Key results
- DocKylin reaches 65.1 on DocVQA val and 77.3 on DocVQA test, leading the general-purpose and text-centric baselines reported in the paper's tables.
- DocKylin with APS+DTS cuts average visual token length on DocVQA from 2770 to 1250 while keeping or improving accuracy (DocVQA val 65.1 vs 63.1 without slimming).
- Applying APS to LLaVA1.5 raises DocVQA from 8.5 to 10.7 and SROIE from 1.7 to 3.7 without retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add APS to LLaVA1.5 (ablation) | LLaVA1.5 · DocVQA val | accuracy: 8.5 -> 10.7 (+27.4%) | Helped |
| add APS to LLaVA1.5 (ablation) | LLaVA1.5 · InfoVQA val | accuracy: 14.7 -> 14.7 (+0%) | No clear effect |
| add APS to Qwen-VL (ablation) | Qwen-VL · SROIE | accuracy: 34.5 -> 40.0 (+15.9%) | Helped |
| add APS to Monkey (ablation) | Monkey · DocVQA val | accuracy: 50.1 -> 56.3 (+12.4%) | Helped |
| add APS to InternVL2 (ablation) | InternVL2 · InfoVQA val | accuracy: 49.5 -> 48.2 | Hurt |
| add APS and resize the processed image back to original resolution (ablation) | InternVL2 · DocVQA val | accuracy: 76.2 -> 77.3 (+1.4%) | Helped |
| add APS and DTS (ablation) | DocKylin · DocVQA val | accuracy: 63.1 -> 65.1 | Helped |
| add APS and DTS (ablation) | DocKylin · ChartQA | accuracy: 55.8 -> 59.0 | Helped |
| add APS and DTS (ablation) | DocKylin · InfoVQA val | accuracy: 34.8 -> 34.8 | No clear effect |
| add APS and DTS (ablation) | DocKylin · POIE | accuracy: 34.5 -> 36.1 | Helped |
| add similarity-weighted aggregation instead of discarding nonessential tokens (ablation) | APS+DTS on LLaVA1.5 · InfoVQA val | accuracy: 14.2 -> 14.9 | Helped |
| add similarity-weighted aggregation instead of discarding nonessential tokens (ablation) | APS+DTS on LLaVA1.5 · SROIE | accuracy: 3.4 -> 3.3 | Hurt |
Surprising findings
- Applying APS directly to high-resolution InternVL2 slightly degrades performance, which the authors attribute to mismatch with its high-resolution training setup; adding resizing recovers gains.
- Reported DTS-mask masking of table regions causes noticeable performance drops on table datasets.
Limitations
- DTS may mistakenly classify some text regions as redundant, especially in table-heavy documents.
- APS struggles with graphics, document borders, table lines and photographs, and has limited effect on natural scene images.
- APS applied directly to high-resolution models like InternVL2 slightly hurts performance.
Experimental setup
Datasets: CASIA-HWDB CCpdf ChartQA DeepForm DocVQA DVQA FigureQA InfoVQA KLC OCRVQA PlotQA PubTabNet
Benchmarks: DocVQA InfoVQA ChartQA FUNSD SROIE POIE VisualMRC DeepForm WTQ TextVQA
Metrics: accuracy ANLS relaxed accuracy CIDEr F1
Baselines: mPLUG-Owl InstructBLIP LLaVA1.5 Qwen-VL Monkey InternVL InternLM-XComposer2 Donut Pix2Struct mPLUG-Doc UReader DocPeida
Builds on: LLaVA Donut PruMerge FastV Qwen-VL
Closest papers by meaning
- LensVLM: Selective Context Expansion for Compressed Visual Representation of Text (NeurIPS 2026 · similarity 0.88)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.88)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.88)
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.88)
- Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models (CVPR 2024 · similarity 0.87)
- Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.