Research map / Vision-language and multimodal / Visual question answering
VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VDocRAG, an image-based RAG framework over visually-rich documents with LVLM retriever/generator, self-supervised compression pre-training (RCR, RCG), and the OpenDocVQA benchmark/dataset.
Problem
Retrieval-augmented question answering over large collections of visually-rich documents (charts, tables, slides, PDFs) where text-based RAG loses information through OCR/parsing.
Approach
Treat documents as images and adapt a large vision-language model into a dual-encoder retriever and generator, using self-supervised pre-training that compresses document image features into the EOS token by aligning them with OCR text via contrastive retrieval (RCR) and generation with a custom attention mask (RCG).
Key results
- VDocRetriever reaches 77.3 nDCG@5 on SlideVQA and 72.9 on InfoVQA in the single-pool setting, versus 53.6 and 49.2 for E5 (the stronger text retriever row in the table).
- VDocRAG achieves 44.2 F1 on SlideVQA and 56.2 ANLS on InfoVQA in the single-pool setting, outperforming text-based RAG (28.6 and 40.5).
- Removing both pre-training tasks (RCG and RCR) lowers InfoVQA retrieval nDCG@5 from 72.9 to 66.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VDocRAG with visual document retrieval instead of text-based RAG | Phi3 RAG · ChartQA single-pool | Relaxed Accuracy: 28.0 -> 52.0 (+24.0) | Helped |
| Use VDocRAG with visual document retrieval instead of text-based RAG | Phi3 RAG · SlideVQA single-pool | F1: 28.6 -> 44.2 (+15.6) | Helped |
| Remove RCR pre-training (ablation) | VDocRetriever · SlideVQA single-pool | nDCG@5: 77.3 -> 75.9 (−1.4) | Hurt |
| Remove RCR pre-training (ablation) | VDocRetriever · InfoVQA single-pool | nDCG@5: 72.9 -> 71.1 (−1.8) | Hurt |
| Remove RCG pre-training (ablation) | VDocRetriever · SlideVQA single-pool | nDCG@5: 77.3 -> 71.7 (−5.6) | Hurt |
| Remove RCG pre-training (ablation) | VDocRetriever · InfoVQA single-pool | nDCG@5: 72.9 -> 68.8 (−4.1) | Hurt |
| Remove both RCG and RCR pre-training tasks (ablation) | VDocRetriever · SlideVQA single-pool | nDCG@5: 77.3 -> 71.0 (−6.3) | Hurt |
| Remove both RCG and RCR pre-training tasks (ablation) | VDocRetriever · InfoVQA single-pool | nDCG@5: 72.9 -> 66.8 (−6.1) | Hurt |
| Remove the LLM and projector, leaving CLIP encoders (ablation) | VDocRetriever · SlideVQA single-pool | nDCG@5: 77.3 -> 43.7 (−33.6) | Hurt |
| Remove the LLM and projector, leaving CLIP encoders (ablation) | VDocRetriever · InfoVQA single-pool | nDCG@5: 72.9 -> 37.9 (−35.0) | Hurt |
| Remove MHDocVQA from the training data (ablation) | VDocRAG · SlideVQA single-pool | nDCG@5: 77.3 -> 75.0 (−2.3) | Hurt |
| Remove all OpenDocVQA datasets except MHDocVQA (ablation) | VDocRAG · InfoVQA single-pool | ANLS: 56.2 -> 44.0 (−12.2) | Hurt |
Surprising findings
- Off-the-shelf text retrievers such as E5-Mistral and NV-Embed-v2 are competitive with or better than some visual retrievers on certain datasets, with BM25 scoring relatively well on DUDE.
Limitations
- OCR-based pre-training limits retrieval of images without text; caption data is suggested as an alternative.
- Creating search indexes for large image collections remains computationally costly.
- Retriever and generator are trained separately rather than jointly; text-heavy documents remain challenging due to OCR limitations.
Experimental setup
Datasets: DocVQA InfoVQA VisualMRC ChartQA OpenWikiTable DUDE MPMQA SlideVQA MHDocVQA DocStruct4M COYO-700M
Benchmarks: ChartQA SlideVQA InfoVQA DUDE OpenDocVQA
Metrics: nDCG@5 ANLS Relaxed Accuracy F1
Baselines: BM25 Contriever E5 GTE E5-Mistral NV-Embed-v2 CLIP DSE VisRAG-Ret Text-based RAG
Builds on: DSE Phi-3-Vision InfoNCE Open-WikiTable DocStruct4M
Closest papers by meaning
- Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines (AAAI 2025 · similarity 0.91)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.90)
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering (NeurIPS 2025 · similarity 0.89)
- VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents (ICLR 2025 · similarity 0.89)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.89)
- SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.