Research map / Large language models / Retrieval and knowledge
VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VisRAG, a pure-vision RAG pipeline where a VLM-based bi-encoder retriever (VisRAG-Ret) embeds document page images directly and a VLM generator (VisRAG-Gen) answers from retrieved page images, avoiding text parsing.
Problem
Text-based RAG requires parsing multi-modal documents into text, losing layout and visual information and introducing parsing errors that degrade retrieval and generation.
Approach
Treat each document page as an image unit, embed it with a VLM bi-encoder (MiniCPM-V 2.0 with position-weighted mean pooling and InfoNCE), and generate answers with VLMs via page concatenation, weighted selection, or native multi-image input, removing the parsing stage entirely.
Key results
- VisRAG-Ret (in-domain) reaches 77.91 average MRR@10, above ColPali (76.54) and MiniCPM (OCR) (74.78).
- VisRAG-Ret trained only on synthetic data gets 71.49 average MRR@10, about 95% of NV-Embed-v2's 75.24 while being trained on far less data.
- End-to-end VisRAG with MiniCPM-V 2.6 generator yields about 40% average relative accuracy gain over the TextRAG pipeline across six datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use vision-based document-page retrieval instead of text-based MiniCPM (OCR) retrieval | VisRAG-Ret, out-of-domain · Six VQA datasets | MRR@10 average: 63.33 -> 71.49 | Helped |
| use vision-based document-page retrieval instead of text-based MiniCPM (OCR) retrieval | VisRAG-Ret, in-domain · Six VQA datasets | MRR@10 average: 74.78 -> 77.91 | Helped |
| use vision-based document-page generation instead of OCR-text generation | VisRAG-Gen vs. TextRAG-Gen · Six VQA datasets, top-1 | accuracy (%) average: 27.68 -> 53.32 | Helped |
| increase retrieved pages from top-1 to top-3 | MiniCPM-V 2.6 multi-image generation · Six VQA datasets | accuracy (%) average: 53.32 -> 56.12 | Helped |
| increase retrieved pages from top-1 to top-3 | GPT-4o multi-image generation · Six VQA datasets | accuracy (%) average: 52.44 -> 54.98 | Helped |
| increase retrieved pages beyond top-1 using page concatenation (ablation) | MiniCPM-V 2.0 single-image generation · Six VQA datasets | accuracy (%) average: 36.16 -> 29.95; 36.16 -> 28.96 | Hurt |
| use weighted selection instead of page concatenation for multiple retrieved pages (ablation) | MiniCPM-V 2.0 single-image generation · Six VQA datasets, top-2 and top-3 | accuracy (%) average: 29.95 -> 36.10; 28.96 -> 35.50 | Helped |
| replace PPOCR adjacent merging with Pytesseract parsing (ablation) | MiniCPM text retriever · Six VQA datasets, in-domain | MRR@10 average: 74.78 -> 66.13 | Hurt |
| replace PPOCR adjacent merging with layout-preserving parsing (ablation) | MiniCPM text retriever · Six VQA datasets, in-domain | MRR@10 average: 74.78 -> 73.55 | Hurt |
| replace VisRAG-Ret with SigLIP (ablation) | VisRAG pipeline · ArxivQA and InfographicsVQA document-length subsets | relative retrieval and generation performance | Hurt |
| replace text parsing and text retrieval with VisRAG vision-based pipeline | VisRAG pipeline · InfographicsVQA | end-to-end accuracy: 25% -> 51% | Helped |
| use VisRAG instead of TextRAG | RAG pipeline · Six evaluation datasets | relative accuracy: 40% relative accuracy increment | Helped |
Surprising findings
- Text embedding model bge-large outperforms BM25 on MTEB but fails on parsed real-world document text in this setting.
- For single-image VLM MiniCPM-V 2.0, adding more retrieved pages (page concatenation or weighted selection) does not improve accuracy.
Limitations
- Relies on a single-image VLM for generation unless multi-image models are used; single-image models do not benefit from more retrieved pages.
- Evaluation focuses on VQA-derived datasets with mostly single-page answers and a relaxed exact match metric.
Experimental setup
Datasets: ArXivQA ChartQA MP-DocVQA InfographicsVQA PlotQA SlideVQA TextVQA ALLaVA
Benchmarks: ArxivQA ChartQA DocVQA InfoVQA PlotQA SlideVQA
Metrics: MRR@10 Recall@10 answer accuracy relaxed exact match
Baselines: BM25 bge-large-en-v1.5 NV-Embed-v2 SigLIP MiniCPM (OCR) MiniCPM (Captioner) ColPali GPT-4o (OCR)
Builds on: DPR-style dual encoder dense retrieval MiniCPM-V 2.0 ColPali InfoNCE
Closest papers by meaning
- SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.90)
- VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents (CVPR 2025 · similarity 0.89)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.89)
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.88)
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token (NeurIPS 2024 · similarity 0.88)
- RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.