Research map / Vision-language and multimodal / Visual question answering
Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DocHaystack and InfoHaystack benchmarks of up to 1,000 document images per question, and V-RAG, a vision-encoder-ensemble retrieval pipeline with an LMM relevance filter feeding an LMM answerer.
Problem
Multi-image VQA benchmarks pair each question with at most ~30 images, which does not reflect large-scale retrieval-and-reasoning over hundreds or thousands of document images; LMMs struggle with such haystack settings.
Approach
Build question-specific document haystacks from DocVQA and InfographicVQA with LLM, manual, and generic-knowledge filtering to ensure unique answers, and retrieve with an ensemble of CLIP, SigLIP, and OpenCLIP similarity followed by an LMM yes/no relevance filter before LMM answering.
Key results
- V-RAG reaches 66.1% Recall@1 on DocHaystack-1000 versus 56.9% for the best baseline (BM25 OCR is the strongest in that table, OpenCLIP/SigLIP ~34-35%), and 64.5% Recall@1 on InfoHaystack-1000 versus 53.6% for OpenCLIP.
- Qwen2-VL fine-tuned with V-RAG reaches 86.2% VQA accuracy on DocHaystack-100 and 73.4% on DocHaystack-1000.
- Adding V-RAG lifts GPT-4o VQA on DocHaystack-100 from 27.5 to 81.7, while raw LLaVA-OV and GPT-4o cannot run at 1000 documents.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use V-RAG instead of the strongest listed retrieval baseline | document retrieval · DocHaystack-1000 | Recall@1: 34.9 -> 66.1 | Helped |
| Use V-RAG instead of the strongest listed retrieval baseline | document retrieval · InfoHaystack-1000 | Recall@1: 53.6 -> 64.5 | Helped |
| Add V-RAG retrieval to GPT-4o | GPT-4o · DocHaystack-200 | accuracy: 23.9 -> 72.5 | Helped |
| Add V-RAG retrieval to GPT-4o | GPT-4o · InfoHaystack-200 | accuracy: 20.0 -> 63.2 | Helped |
| Use three vision encoders instead of CLIP alone (ablation) | V-RAG retrieval · DocHaystack-1000 | Recall@1: 23.9 -> 42.2 | Helped |
| Use three vision encoders instead of CLIP alone (ablation) | V-RAG retrieval · InfoHaystack-1000 | Recall@1: 45.8 -> 56.1 | Helped |
| Add the LMM-filter to the three-encoder ensemble (ablation) | V-RAG retrieval · DocHaystack-1000 | Recall@1: 42.2 -> 66.1 | Helped |
| Add the LMM-filter to the three-encoder ensemble (ablation) | V-RAG retrieval · InfoHaystack-1000 | Recall@1: 56.1 -> 64.5 | Helped |
| Increase LMM-VQA selection from Top 1 to Top 5 (ablation) | Qwen2-VL with V-RAG · DocHaystack-1000 | accuracy: 60.7 -> 66.1 | Helped |
| Increase LMM-VQA selection from Top 1 to Top 5 (ablation) | LLaVA-OV with V-RAG · DocHaystack-1000 | accuracy: 40.7 -> 21.9 | Hurt |
Surprising findings
- BM25 over OCR text outperforms all text-to-image retrievers (including V-RAG's components) on DocHaystack-1000 in the baseline table, with V-RAG 66.1 vs BM25 56.9.
- About 27% of DocVQA and 56% of InfographicVQA questions can be answered by GPT-4o without the image, motivating generic-knowledge filtering.
Limitations
- Benchmark questions derive from DocVQA and InfographicVQA, and filtering relies on GPT-4o and manual review.
- Raw LMMs such as LLaVA-OV and GPT-4o cannot process 1000-document haystacks directly due to context limits, requiring retrieval first.
- Ablation of top-k shows LLaVA-OV degrades at top-5, indicating difficulty with multiple images.
Experimental setup
Datasets: DocVQA InfographicVQA
Benchmarks: DocHaystack-100 DocHaystack-200 DocHaystack-1000 InfoHaystack-100 InfoHaystack-200 InfoHaystack-1000
Metrics: Recall@1 Recall@3 Recall@5 VQA accuracy
Baselines: BM25 (OCR) Jina-CLIP Nomic-Embed-Vision CLIP SigLIP OpenCLIP LLaVA-OneVision GPT-4o Gemini Qwen2-VL MIRAGE
Builds on: RetVQA MIRAGE DocVQA InfographicVQA
Closest papers by meaning
- Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark (ICLR 2025 · similarity 0.94)
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.91)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.91)
- SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.91)
- VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents (CVPR 2025 · similarity 0.90)
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.