Research map / Vision-language and multimodal / Multimodal LLMs
MMPB: It’s Time for Multi-Modal Personalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMPB, a 10k-query multiple-choice benchmark for evaluating vision-language models on personalization across concept injection, multi-turn dialogue, and personalized querying, finding most of 23 VLMs struggle, especially closed models that refuse human-related queries.
Problem
Current VLM benchmarks do not evaluate whether vision-language models can adapt to individual users' visual concepts and preferences, especially in cold-start settings with minimal context and over multi-turn dialogue.
Approach
A benchmark that injects personalizable concepts (111 across humans, animals, objects, characters) via text (four granularity levels) or images (1, 2, 5 references), then tests awareness, appropriateness, and coherency queries (including preference-grounded VQA) after 0 or 10 dialogue turns, with quality control against concept-only and image-only solvability.
Key results
- Text-based concept injection averages 63.8% vs 57.8% for image-based in the 0-turn setting; 57.9% vs 52.6% at 10 turns, with human evaluators at 92.1%.
- Closed-source models average 51.4% vs 59.9% for open-source across all settings, with up to 2,237 of 7,501 human-category queries triggering refusals.
- Preference-grounded VQA is much harder than recognition; models strong on general VQA consistently struggle with preference queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use text-based rather than image-based concept injection | VLMs evaluated on MMPB · MMPB, 0-turn | accuracy: 63.8% vs. 57.8% | Helped |
| Use text-based rather than image-based concept injection | VLMs evaluated on MMPB · MMPB, 10-turn | accuracy: 57.9% vs. 52.6% | Helped |
| Add a 10-turn conversation before personalized querying | VLMs evaluated on MMPB · MMPB | accuracy | Hurt |
| Use open rather than closed models | VLMs evaluated on MMPB · MMPB, averaged across experimental settings | accuracy: 51.4% vs. 59.9% | Helped |
| Use a detailed rather than moderate text description (ablation) | MMPB recognition evaluation · MMPB recognition, 0-turn, text injection | accuracy: 77.1 { $\pm$ 6.7} -> \textbf{79.3} { $\pm$ 6.7} | Helped |
| Use a detailed rather than moderate text description (ablation) | MMPB recognition evaluation · MMPB recognition, 10-turn, text injection | accuracy: 72.0 { $\pm$ 7.2} -> \textbf{73.5} { $\pm$ 7.9} | Helped |
| Use simple keywords rather than a moderate text description (ablation) | MMPB recognition evaluation · MMPB recognition, 10-turn, text injection | accuracy: 72.0 { $\pm$ 7.2} vs. 67.4 { $\pm$ 4.7} | Hurt |
| Use five rather than two reference images (ablation) | MMPB recognition evaluation · MMPB recognition, 0-turn, image injection | accuracy: 72.2 { $\pm$ 9.9} -> \textbf{73.3} { $\pm$ 10.5} | Helped |
| Use five rather than two reference images (ablation) | MMPB recognition evaluation · MMPB recognition, 10-turn, image injection | accuracy: 69.2 { $\pm$ 8.4} -> \textbf{70.1} { $\pm$ 9.4} | Helped |
| Use detailed rather than moderate text descriptions in multi-concept inputs (ablation) | VLMs evaluated on MMPB · MMPB multi-concept, target near the middle | accuracy | Hurt |
| Use image-based rather than text-based injection for animal hard negatives | VLMs evaluated on MMPB · MMPB animal Appropriateness, same-species hard negatives | accuracy | Helped |
| Use multi-entity rather than single-entity query images | VLMs evaluated on MMPB · MMPB Coherency | accuracy | Hurt |
Surprising findings
- Closed-source models such as GPT-4o and Claude refuse identity-related queries, and InternVL2.5-78B ranks first on general VQA yet performs weakly on personalization.
- Simple three-keyword text injection achieves accuracy comparable to a single reference image in the 10-turn setting, and five images do not exceed moderate text descriptions.
Limitations
- Assumes static concept appearance and preferences; does not model evolving user traits.
- Evaluation limited to multiple-choice/open-ended VQA on awareness, appropriateness, coherency; does not cover applications like personalized captioning or robotics.
- Human baseline computed on a five-concept subset rather than the full benchmark.
Experimental setup
Datasets: MC-LLaVA images MyVLM images LMSYS-Chat-1M
Benchmarks: MMPB
Metrics: accuracy
Baselines: GPT-4o Claude-3.5-Sonnet Claude-3.7-Sonnet Gemini-1.5-Flash Gemini-2.0-Flash InternVL2.5-78B Qwen2.5-VL-72B LLaVA-NeXT Ovis2-34B DeepSeek-VL2
Builds on: MyVLM Yo'LLaVA MC-LLaVA LMSYS-Chat-1M
Closest papers by meaning
- Ego: Embedding-Guided Personalization of Vision-Language Models (CVPR 2026 · similarity 0.89)
- Personalized Visual Instruction Tuning (ICLR 2025 · similarity 0.89)
- PersonaVLM: Long-Term Personalized Multimodal LLMs (CVPR 2026 · similarity 0.89)
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.89)
- Contextualized Visual Personalization in Vision-Language Models (ICML 2026 · similarity 0.89)
- Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.