Research map / Vision-language and multimodal / Visual question answering
SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SV-RAG, a single MLLM with two LoRA adapters (one for contextualized late-interaction page retrieval, one for question answering) to answer questions over long visually-rich documents, and introduces VisR-Bench.
Problem
Answering questions over long, multi-page visually-rich documents is hard for MLLMs: parsing loses layout and feeding all pages is memory-expensive and distracting, while document-parser RAG underperforms.
Approach
Use the MLLM itself as a visual retriever via contextualized late-interaction scoring on hidden states, and share one backbone across retrieval and QA through two LoRA adapters, so only the top-k retrieved pages are passed to the QA adapter.
Key results
- On SlideVQA single-page retrieval, Col-Phi-3-vision reaches 90.6 top-1 and 98.8 top-5, above the best text baseline NV-Embed-v2 (82.2 top-1).
- SV-RAG InternVL2-4B with fine-tuned QA adapter reaches 23.0 G-Acc on MMLongBench-Doc, a 4B model that the authors report as comparable to Claude-3 Opus and Gemini-1.5-Pro in their table.
- On SP-SlideVQA, the fine-tuned PaliGemma QA adapter raises EM from 35.03 to 49.75 and fine-tuned Phi-3-vision reaches 58.13 EM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Col-Phi-3-vision for page retrieval instead of the strongest listed baseline | SV-RAG retrieval · SlideVQA | top1 accuracy: 82.2 -> 90.6 | Helped |
| use Col-Phi-3-vision for page retrieval instead of the strongest listed baseline | SV-RAG retrieval · MMLongBench-Doc | top1 accuracy: 44.9 -> 64.8 | Helped |
| use Col-Phi-3-vision for page retrieval instead of the strongest listed baseline | SV-RAG retrieval · VisR-Bench | top1 accuracy: 55.2 -> 71.9 | Helped |
| use Col-Phi-3-vision for page retrieval instead of the strongest listed baseline | SV-RAG retrieval · SP-DocVQA | top1 accuracy: 56.7 -> 65.1 | Helped |
| add QA LoRA fine-tuning (ablation) | SV-RAG-InternVL2 · SP-SlideVQA | EM: 16.40 -> 45.07 | Helped |
| add QA LoRA fine-tuning (ablation) | SV-RAG-InternVL2 · MMLongBench-Doc | G-Acc: 33.2 -> 34.0 | Helped |
| add QA LoRA fine-tuning (ablation) | SV-RAG-InternVL2 · MP-SlideVQA | EM: 24.25 -> 31.98 | Helped |
| add QA LoRA fine-tuning (ablation) | SV-RAG-InternVL2 · DUDE | ANLS: 0.36 -> 0.45 | Mixed |
| add QA LoRA fine-tuning (ablation) | SV-RAG-Phi-3-vision · MMLongBench-Doc | G-Acc: 30.7 -> 28.4 | Hurt |
| use hidden states from the 21th transformer layer instead of the final layer (ablation) | Phi-3-vision retrieval · MMLongBench-Doc | top1 accuracy | Helped |
| expand retrieval training data with DocMatix-IR and PFL-DocVQA | SV-RAG retrieval · MMLongBench-Doc | top1 retrieval accuracy: ~1% | Helped |
Surprising findings
- Fine-tuning the QA adapter on SlideVQA reduced MMLongBench-Doc G-Acc for the Phi-3-vision backbone (30.7 to 28.4), suggesting fine-tuning can hurt generalization.
- Giving GPT-4o only the ground-truth evidence page slightly beats giving it all pages, indicating distraction from irrelevant pages in long contexts.
Limitations
- Requires computational resources for training and inference, limiting resource-constrained use.
- Not evaluated on real on-device hardware despite mobile-friendly claims.
- Retrieval accuracy drops substantially on longer, more complex documents such as MMLongBench-Doc and VisR-Bench.
Experimental setup
Datasets: ColPali training data DocVQA InfoVQA TAT-DQA arXivQA DocMatix-IR PFL-DocVQA SlideVQA
Benchmarks: SlideVQA MMLongBench-Doc DocVQA DUDE VisR-Bench
Metrics: Top-k accuracy EM G-Acc ANLS PNLS MGS
Baselines: BM25 SBERT BGE-M3 BGE-large NV-Embed-v2 CLIP SigLIP InternVL2-8B GPT-4o Gemini-1.5-Pro
Builds on: ColPali LoRA LLaVA ColBERT-style late interaction
Closest papers by meaning
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.91)
- VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents (ICLR 2025 · similarity 0.90)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.89)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.89)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.88)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.