Research map / Vision-language and multimodal / Human and embodied modeling

ColPali: Efficient Document Retrieval with Vision Language Models

ICLR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Composed and text-based image retrieval · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the ViDoRe benchmark for visually rich document retrieval and ColPali, a PaliGemma-3B-based late-interaction retriever that embeds document page images directly, outperforming OCR/captioning text pipelines.

Problem

Document retrieval pipelines rely on lengthy PDF parsing, OCR, layout detection, and captioning to index visually rich documents, and no benchmark evaluates end-to-end page-level retrieval using both textual and visual features.

Approach

Index document pages purely from their images by feeding page patches into a VLM (PaliGemma-3B) to produce ColBERT-style multi-vector embeddings (dimension 128), trained end-to-end with an in-batch contrastive late-interaction loss.

Key results

• ColPali reaches 81.3 average nDCG@5 on ViDoRe versus 58.8 for BiPali and 67.0 for the best Unstructured + captioning BGE-M3 pipeline.
• ColPali indexes pages at 0.39 s per page versus 7.22 s for the Unstructured pipeline on an NVIDIA L4, and query latency is about 30 ms versus 22 ms for BGE-M3.
• Token pooling with pool factor 3 cuts stored vectors by 66.7% while keeping 97.8% of performance; ColQwen2 (768 patches) reaches 86.6 average nDCG@5 in the ablations.

Measured effects reported in the paper

Surprising findings

• Fine-tuning SigLIP into a late-interaction model (ColSigLIP) yields abysmal performance (3.2 avg nDCG@5), while the PaliGemma-based ColPali works well.
• A ColPali variant trained only on DocMatix (no in-domain ViDoRe train data) loses only 2.2 nDCG@5, still beating the closest baseline by over 12 points.

Limitations

• Storage footprint is an order of magnitude larger than BM25 and two orders larger than BGE-M3 unless compressed.
• Practical benchmark queries were generated by Claude-3 Sonnet and some academic datasets' train sets overlap with training data, so results may be partly in-distribution.
• Query encoding with the VLM is slower than a standard text encoder (about 30 ms vs 22 ms).

Experimental setup

Datasets: DocVQA InfoVQA TAT-DQA arXivQA Scrapped PDFs (synthetic, Claude-3 Sonnet queries) DocMatix TabFQuAD

Benchmarks: ViDoRe

Metrics: nDCG@5 Recall@1 MRR Recall@K query latency indexing latency embedding size

Baselines: BM25 BGE-M3 Unstructured (text-only) Unstructured + OCR Unstructured + captioning Jina-CLIP Nomic-vision SigLIP BiSigLIP BiPali

Builds on: ColBERT PaliGemma SigLIP BGE-M3 Idefics2

Closest papers by meaning

• Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark (CVPR 2025 · similarity 0.90)
• SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.87)
• Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.87)
• CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.87)
• VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents (ICLR 2025 · similarity 0.86)
• EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models (AAAI 2025 · similarity 0.86)