Research map / Vision-language and multimodal / Multimodal LLMs
Docopilot: Improving Multimodal Models for Document-Level Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Doc-750K, a 758K-QA multi-page document dataset, and Docopilot, a retrieval-free MLLM fine-tuned from InternVL2 on it that handles cross-page dependencies natively.
Problem
Multimodal LLMs perform poorly on complex multi-page document understanding due to a lack of high-quality document-level training data, and RAG pipelines suffer from fragmented context, error accumulation, and latency.
Approach
Build a large document-level multimodal QA dataset from academic papers (OpenReview, arXiv, Sci-Hub) with real and structure-derived QA pairs, then fine-tune InternVL2 on it with efficiency tools (multimodal packing, Ring Attention, Liger Kernel) so the model handles long multi-page context directly without retrieval.
Key results
- Docopilot-8B scores 61.8 on MM-NIAH overall vs 42.9 for InternVL2-8B, and 52.8 for InternVL2-26B.
- Docopilot-8B reaches 28.8% accuracy on MMLongBench-Doc vs 17.4% for InternVL2-8B and 24.2% for InternVL2-8B + RAG.
- Docopilot-2B gets 21.8% accuracy on MMLongBench-Doc vs 10.5% for InternVL2-2B; RAG-based InternVL2-2B adds 82.9ms latency vs 35.9ms for Docopilot-2B with the same base latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune InternVL2-8B on Doc-750K and use native long-document input instead of RAG | InternVL2-8B / Docopilot-8B · MMLongBench-Doc | Acc: 24.2 -> 28.8 | Helped |
| fine-tune InternVL2-8B on Doc-750K and use native long-document input instead of RAG | InternVL2-8B / Docopilot-8B · MMLongBench-Doc | F1: 24.5 -> 23.0 | Hurt |
| fine-tune InternVL2-8B on Doc-750K for long-context understanding | InternVL2-8B / Docopilot-8B · MM-NIAH | Overall: 42.9 -> 61.8 | Helped |
| add Sci-Hub data to the SFT recipe without Doc-750K (ablation) | Variant1 SFT recipe · MMLongBench-Doc | F1: 9.4 -> 15.2 | Helped |
| add Arxiv data to the SFT recipe (ablation) | Variant2 SFT recipe · MMLongBench-Doc | Acc: 18.5 -> 20.5 | Helped |
| add Arxiv data to the SFT recipe (ablation) | Variant2 SFT recipe · MMLongBench-Doc | F1: 15.2 -> 15.5 | Helped |
| add OpenReview data to the SFT recipe (ablation) | Variant3 SFT recipe · MMLongBench-Doc | Acc: 20.5 -> 21.8 | Helped |
| add OpenReview data to the SFT recipe (ablation) | Variant3 SFT recipe · MMLongBench-Doc | F1: 15.5 -> 16.0 | Helped |
| use Docopilot-8B rather than InternVL2-8B | InternVL2-8B · DocVQA | ANLS: 91.6 -> 92.0 | Helped |
| use Docopilot-8B rather than InternVL2-8B | InternVL2-8B · ChartQA | Relaxed EM: 83.3 -> 83.3 | No clear effect |
| use Docopilot-8B rather than InternVL2-8B | InternVL2-8B · InfoVQA | ANLS: 74.8 -> 73.3 | Hurt |
Surprising findings
- Docopilot-8B's MM-NIAH overall score is 61.8 vs 52.8 for the larger InternVL2-26B, despite the smaller size.
- Pure-text long-context datasets were included in the SFT mix, and the RAG baseline is outperformed even though retrieval adds latency.
Limitations
- Doc-750K is restricted to academic documents.
- Only about 4.8% of the data is LLM-generated, and the evaluation relies on GPT-4o as judge for open-ended QA.
- Single-page results are comparable rather than improved over InternVL2 (e.g., InfoVQA drops for Docopilot-8B).
Experimental setup
Datasets: Doc-750K MP-Docmatix MP-DocVQA DUDE Taesiri-ArxivQA MMDU-45K DocVQA DocReason InfoVQA ChartQA LongAlpaca LongCite
Benchmarks: MP-DocVQA MMLongBench-Doc DocGenome MM-NIAH DocVQA ChartQA InfoVQA
Metrics: ANLS Accuracy F1 Relaxed EM GPT accuracy latency
Baselines: InternVL2-2B InternVL2-8B InternVL2-26B InternVL2-8B + RAG VisRAG M3DocRAG MiniCPM-V2.6 Gemini-1.5-Pro GPT-4o
Builds on: InternVL2 Ring Attention Liger Kernel MMLongBench-Doc
Closest papers by meaning
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.90)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.90)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.89)
- SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.89)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.89)
- DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.