Research map / Vision-language and multimodal / Human and embodied modeling
Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NL-DIR, a 41K-document-image benchmark with 205K LLM-generated and human-verified natural-language queries, and evaluates contrastive VLMs and OCR-free VDU models in a two-stage recall plus cross-attention re-ranking pipeline.
Problem
Existing document image retrieval relies on image queries that match only coarse categories or on OCR-based text retrieval, leaving fine-grained natural-language queries over document images largely unsupported.
Approach
Builds a large natural-language query-to-document-image retrieval benchmark by generating queries from layout-aware OCR text with an LLM, filtering with CLIP/BLIP/ChatGPT/Qwen-VL scores plus manual checks, and proposes a two-stage recall (SigLIP, LoRA-tuned) and cross-attention re-ranking (BLIP-ITM/Pix2Struct) pipeline.
Key results
- SigLIP-So400m zero-shot reaches Recall@10 of 61.18; fine-tuning SigLIP image encoder with LoRA raises Recall@10 to 89.72 in the recall stage.
- Two-stage pipeline with BLIP-ITM text encoder and Pix2Struct visual encoder with extra cross-attention reaches Recall@1 81.03 and MRR@10 85.68 after re-ranking, versus 76.29 MRR@10 for recall alone.
- Best generative VDU model zero-shot (Qwen2-VL) reaches only Recall@10 of 1.66, far below contrastive VLMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the SigLIP text module with LoRA (ablation) | SigLIP-So400m on NL-DIR · NL-DIR test | Recall@10: 61.18 -> 79.40 | Helped |
| Fine-tune the SigLIP visual module with LoRA (ablation) | SigLIP-So400m on NL-DIR · NL-DIR test | Recall@10: 79.40 -> 89.72 | Helped |
| Use the Pix2Struct visual encoder with the LoRA-tuned SigLIP text encoder (ablation) | SigLIP-So400m on NL-DIR · NL-DIR test | Recall@10: 79.40 -> 41.57 | Hurt |
| Add extra cross-attention to SigLIP encoders (ablation) | SigLIP-So400m reranking · NL-DIR test | Recall@10: 89.72 -> 12.98 | Hurt |
| Use Pix2Struct visual features with extra cross-attention (ablation) | SigLIP text encoder on NL-DIR · NL-DIR test | Recall@10: 12.98 -> 79.40 | Helped |
| Use Pix2Struct's original cross-attention for the ITM task (ablation) | Pix2Struct on NL-DIR · NL-DIR test | Recall@10: 79.40 -> 84.63 | Helped |
| Use BLIP-ITM text features with Pix2Struct visual features and extra cross-attention (ablation) | NL-DIR two-stage retrieval · NL-DIR test | Recall@10: 84.63 -> 94.17 | Helped |
| Add fine-grained reranking (ablation) | NL-DIR retrieval · NL-DIR abstract queries | Recall@10: 87.76 -> 90.56 | Helped |
| Add fine-grained reranking (ablation) | NL-DIR retrieval · NL-DIR concrete queries | Recall@10: 97.22 -> 99.35 | Helped |
| Add fine-grained reranking (ablation) | NL-DIR retrieval · NL-DIR chart subset | Recall@10: 99.02 -> 98.65 | Hurt |
| Add fine-grained reranking (ablation) | NL-DIR retrieval · NL-DIR advertisement subset | Recall@10: 99.01 -> 97.05 | Hurt |
| Remove pairwise loss from reranking training (ablation) | NL-DIR reranking model · NL-DIR test | retrieval performance | Hurt |
Surprising findings
- Zero-shot generative VDU models perform near chance (Recall@10 below 2%), far below contrastive VLMs.
- Re-ranking with a BLIP-ITM text encoder and SigLIP visual encoder degrades performance below the recall stage (Recall@1 13.85 vs 69.33 recall-stage baseline).
Limitations
- Queries generated by ChatGPT with hallucination risk, mitigated only by scoring and manual filtering.
- Re-ranking with cross-attention degrades on advertisement-style documents and some baselines show negative optimization.
- Fine-tuning DSE on the training set transferred poorly, and larger visual encoders increased encoding time without gains.
Experimental setup
Datasets: OCR-IDL DocVQA NL-DIR
Benchmarks: NL-DIR
Metrics: Recall@1 Recall@10 MRR@10 Recall@100
Baselines: CLIP-base BLIP-base BLIP-large DFN SigLIP-So400m InternVL-14B-224px Donut Nougat Pix2Struct Vary TextMonkey DocOwl1.5
Builds on: CLIP BLIP SigLIP ColPali DSE
Closest papers by meaning
- ColPali: Efficient Document Retrieval with Vision Language Models (ICLR 2025 · similarity 0.90)
- INQUIRE: A Natural World Text-to-Image Retrieval Benchmark (NeurIPS 2024 · similarity 0.88)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.87)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.86)
- MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions (ICML 2024 · similarity 0.86)
- MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.