Research map / Vision-language and multimodal / Human and embodied modeling
Generative Multi-Modal Knowledge Retrieval with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GeMKR, an end-to-end generative multi-modal knowledge retriever that uses LLaMA with object-aware prefix-tuning on CLIP and knowledge-clue constrained decoding over an FM-Index to retrieve documents from image+question queries.
Problem
Knowledge retrieval with multi-modal (image+text) queries is ineffective and training-inefficient when built from multiple separate uni-modal retrievers, and existing generative retrieval uses static identifiers poorly suited to multi-aspect, multi-modal knowledge.
Approach
Generate knowledge clues (document substrings unique to one document) with an LLM conditioned on visual and textual queries, then look them up in an FM-Index database; visual features come from CLIP with object-aware prefix tuning and dual-flow attention, and decoding is constrained so clues map to exactly one document.
Key results
- On OKVQA-WK21M (21M-document KB), GeMKR reaches P@5 46.0 vs 31.4 for the best baseline ReViz-ICT.
- On OKVQA-GS112K, GeMKR achieves P@5 49.1, R@5 78.6, R@10 86.2, improving over ReViz-ICT (41.7/73.4/83.2).
- On ReMuQ, GeMKR reaches P@1 75.2 and R@5 90.3 versus 62.1 and 76.2 for ReViz-ICT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GeMKR instead of ReViz-ICT | OKVQA-GS112K retrieval · OKVQA-GS112K | P@5: 41.7 -> 49.1 | Helped |
| use GeMKR instead of ReViz-ICT | OKVQA-WK21M retrieval · OKVQA-WK21M | P@5: 31.4 -> 46.0 | Helped |
| use GeMKR instead of ReViz-ICT | ReMuQ retrieval · ReMuQ | P@1: 62.1 -> 75.2 | Helped |
| add object features (ablation) | GeMKR · OKVQA-GS112K | P@5: 46.9 -> 49.1 | Helped |
| use dual-flow attention instead of vanilla attention (ablation) | GeMKR · OKVQA-GS112K | P@5: 47.5 -> 49.1 | Helped |
| add object-aware prefix tuning (ablation) | GeMKR · OKVQA-GS112K | P@5: 46.5 -> 49.1 | Helped |
| add LoRA adaptation (ablation) | GeMKR · OKVQA-GS112K | P@5: 44.5 -> 49.1 | Helped |
| add object-aware prefix tuning and LoRA adaptation (ablation) | GeMKR · OKVQA-GS112K | P@5: 39.5 -> 49.1 | Helped |
| include the visual module instead of using textual queries and image captions (ablation) | GeMKR · OKVQA-GS112K | P@5: 40.4 -> 49.1 | Helped |
| generate knowledge clues with constraints rather than full documents, first sentences, or free text (ablation) | GeMKR decoding · OKVQA-GS112K | R@5: 51.6; 62.4; 78.6; 64.5 | Helped |
Surprising findings
- Removing both object-aware prefix tuning and LoRA (training only projection layers) drops performance below some baselines, showing the LLM's adaptation is essential.
- Using only image captions as visual input still outperforms the best textual generative baseline SEAL.
Limitations
- Relies on a 7B-13B LLM and CLIP backbone; scaling beyond LLaMA-13B was not explored due to compute cost.
- Knowledge clues mapping to multiple documents are discarded as invalid outputs.
- Evaluated only on VQA-derived retrieval benchmarks with image+question queries.
Experimental setup
Datasets: OKVQA-GS112K OKVQA-WK21M ReMuQ
Benchmarks: OKVQA-GS112K OKVQA-WK21M ReMuQ
Metrics: P@5 P@1 R@5 R@10
Baselines: BM25 DPR CorpusBrain SEAL CLIP VRR ReViz ReViz-ICT
Builds on: SEAL CorpusBrain DSI ReViz LLaMA
Closest papers by meaning
- Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines (AAAI 2025 · similarity 0.89)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.88)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.88)
- Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering (NeurIPS 2025 · similarity 0.88)
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.88)
- CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoning (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.