Research map / Vision-language and multimodal / Multimodal LLMs
VL-DocIR: A Benchmark for Vision-Based Long Document Retrieval
Simon Ott, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiale Wei, Di Wen, Kunyu Peng, Jiaming Zhang, Rainer Stiefelhagen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VL-DocIR, a page-level benchmark for evidence-complete vision-based long document retrieval where answers may span multiple pages or documents.
Problem
Current vision-based document retrieval evaluations emphasize short documents and single relevant pages, missing realistic long-document retrieval where evidence may span multiple pages or documents.
Approach
A page-level benchmark targeting evidence-complete retrieval over long visual documents, requiring all evidence pages needed to answer a query to be retrieved rather than a single relevant page.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-vector instead of single-vector retrieval | Jina V4 · VL-DocIR Macro | nDCGAll@10: 48.03 -> 59.59 | Improved |
| use multi-vector instead of single-vector retrieval | Jina V4 · VL-DocIR Macro | RecallAll@10: 60.24 -> 70.77 | Improved |
| add document-context scoring (ablation) | ColPali v1.3 · VL-DocIR Macro | RecallAll@10: 49.13 -> 65.69 | Improved |
| add document-context scoring (ablation) | ColQwen2.5 · VL-DocIR Macro | RecallAll@10: 67.23 -> 75.78 | Improved |
| add reranking with Jina Reranker M0 (ablation) | ColNomic · VL-DocIR Macro | RecallAll@10: 71.76 -> 74.62; nDCGAll@10 61.82 -> 65.00 | Mixed |
| add query splitting (ablation) | ColQwen2.5 · VL-DocIR Macro | RecallAll@10: 67.23 -> 69.19 | Improved |
| fuse ColNomic (Ctx) with OCR-BM25 via RRF (ablation) | ColNomic (Ctx) · VL-DocIR Macro | nDCGAll@10 / RecallAll@10: RecallAll 78.19 -> 78.67; nDCGAll 66.03 -> 65.13 | Mixed |
| ensemble base, Ctx and rerank lists (ablation) | ColNomic · VL-DocIR Macro | nDCGAll@10: 66.03 -> 69.28 | Improved |
| move from within-document to cross-document evidence | ColNomic B+Ctx+Rank · evidence structure | RecallAll@5: 66.3 -> 27.6 (Doc-Far -> Cross-Doc) | Hurt |
| apply document-context scoring on Micro nDCGAll for ColNomic (ablation) | ColNomic · VL-DocIR Micro | nDCGAll@10: 70.68 -> 70.07 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- ColPali v1.3 — ColPali: Efficient Document Retrieval with Vision Language Models (ICLR 2025) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Benchmarks: VL-DocIR
Closest papers by meaning
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.89)
- ColPali: Efficient Document Retrieval with Vision Language Models (ICLR 2025 · similarity 0.89)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.89)
- DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding (CVPR 2026 · similarity 0.88)
- TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning (NeurIPS 2026 · similarity 0.88)
- Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
