Research map / Vision-language and multimodal / Multimodal LLMs
RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAP, a retrieval-augmented framework that lets MLLMs personalize to user concepts via an editable external database, trained with a 260K-sample personalization dataset to avoid per-concept fine-tuning.
Problem
MLLMs lack user-specific knowledge, and existing personalization methods (MyVLM, Yo'LLaVA, fine-tuning) need multiple labeled images and model updates for each new concept.
Approach
Store user concepts as image-text key-value entries in a database; detect regions of interest with an open-vocabulary detector, retrieve matching concepts with CLIP-based FAISS search, and feed their information into the MLLM, which is instruction-tuned on a custom dataset with noise-concept negatives so it generalizes to new concepts without retraining.
Key results
- RAP-LLaVA reaches 93.51 recall, 96.47 precision and 94.97 F1 on personalized captioning, versus 87.82 F1 for LLaVA-LoRA and 85.50 for MyVLM-LLaVA.
- RAP-LLaVA achieves 0.936 weighted accuracy on personalized QA and 0.980 on visual recognition, the best among compared open methods, and is comparable to GPT-4V (0.924 and 0.901 with one image).
- With a 500-concept database, Top-5 retrieval recall still exceeds 90%, per the appendix.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAP-LLaVA instead of the strongest listed captioning baseline | Personalized image captioning · Personalized image captioning | F1-score: 87.82 -> 94.97 | Helped |
| Use RAP-LLaVA instead of the strongest listed non-RAP personalized QA baseline | Personalized question answering · Personalized question answering | Weighted accuracy: 0.906 -> 0.936 | Helped |
| Add an external knowledge base | RAP-LLaVA · InfoSeek | score: 0.218 -> 0.344 | Helped |
| Skip retrieval and provide relevant concept information directly (ablation) | RAP-LLaVA · Personalized image captioning | Precision: 96.47 -> 100.0 | Helped |
| Add text information (ablation) | RAP-LLaVA · Personalized image captioning | F1-score: 91.68 -> 94.97 | Helped |
| Add data augmentation (ablation) | RAP-LLaVA · Personalized image captioning | F1-score: 93.42 -> 94.97 | Helped |
| Add negative samples (ablation) | RAP-LLaVA · Personalized image captioning | F1-score: 72.40 -> 94.97 | Helped |
| Add text information (ablation) | RAP-LLaVA · Personalized question answering | Weighted accuracy: 0.921 -> 0.936 | Helped |
| Add data augmentation (ablation) | RAP-LLaVA · Personalized question answering | Weighted accuracy: 0.921 -> 0.936 | Helped |
| Add negative samples (ablation) | RAP-LLaVA · Personalized question answering | Weighted accuracy: 0.925 -> 0.936 | Helped |
| Add text information (ablation) | RAP-LLaVA · Personalized visual recognition | Weighted accuracy: 0.959 -> 0.980 | Helped |
| Add data augmentation (ablation) | RAP-LLaVA · Personalized visual recognition | Weighted accuracy: 0.965 -> 0.980 | Helped |
Surprising findings
- Removing the negative-sample training data drops precision from 96.47 to 58.21 F1-relevant precision collapse, showing negatives are critical for filtering retrieval noise.
- The vanilla LLaVA with retriever barely improves captioning recall (1.26), indicating that retrieved concept information alone is not used without the proposed training.
Limitations
- Retrieval accuracy drops as the database grows, lowering precision for larger concept sets.
- Limited context length restricts how many retrieved concepts can be included (2 or 3).
Experimental setup
Datasets: RefCOCO ILSVRC2015-VID TAO CustomConcept101 Object365 CelebA LLaVA-Instruct-665K
Benchmarks: MyVLM personalized captioning dataset Yo'LLaVA personalized QA benchmark MMMU InfoSeek
Metrics: recall precision F1-score accuracy BLEU CIDEr
Baselines: LLaVA LLaVA-LoRA MyVLM Yo'LLaVA GPT-4V
Builds on: LLaVA MyVLM Yo'LLaVA YOLO-World LoRA
Closest papers by meaning
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.89)
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.88)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- PersonaVLM: Long-Term Personalized Multimodal LLMs (CVPR 2026 · similarity 0.88)
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.