Research map / Vision-language and multimodal / Multimodal LLMs

Contextualized Visual Personalization in Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes contextualized visual personalization and proposes CoViP, which RL-post-trains Qwen3-VL-8B on a synthetic personalized image captioning benchmark (with F1 recognition and MCQA retrieval rewards) and uses caption-augmented generation at inference, improving captioning and three diagnostic personalization tasks.

Problem

VLMs fail to generate personalized responses grounded in a user's accumulated visual-textual interaction history, and existing personalization methods focus on explicit attributes rather than implicit, experience-level context.

Approach

Treat personalized image captioning as a proxy task for contextualized visual personalization: RL post-training with a set-level F1 recognition reward and a contrastive MCQA-based retrieval reward, then reuse the model's own caption as a conditioning signal for downstream responses (caption-augmented generation).

Key results

• On CapEval-QAs, Qwen3-VL-8B + CoViP reaches 77.4 Acc+ on 1-concept vs 39.0 for base Qwen3-VL-8B (+38.4) and 58.1 for Gemini-3.0 Pro.
• On ITR with CAG, CoViP achieves 42.8 recall vs 9.4 for base Qwen3-VL-8B (direct) and 19.0 for Gemini-3.0 Pro with CAG.
• CoViP improves average multi-image benchmark accuracy from 65.5% to 67.7% across MM-NIAH, MMNeedle, MuirBench and MMIU.

Surprising findings

• RAP, a post-training personalization baseline, underperforms the zero-shot Qwen3-VL-8B on the captioning benchmark.
• CAG substantially hurts ITR for proprietary models (e.g., Gemini-2.0-Flash drops from 66.1 to 12.2 recall) due to generic captions lacking fine-grained trigger details.

Limitations

• Benchmark relies on synthetic dialogues and generated images that may contain factual inconsistencies or visual artifacts.
• Evaluation does not cover real-world, noisy, heterogeneous multi-source user context.
• Possible textual shortcut learning limits robustness to conflicting or noisy real-world memory.

Experimental setup

Datasets: MMPB Yo'LLaVA CelebA MyVLM Custom101 RAP-LLaVA DreamBooth Unsplash Pexels

Benchmarks: CapEval-QAs (personalized image captioning benchmark) LSD LAR ITR MM-NIAH MMNeedle MuirBench MMIU

Metrics: Positive accuracy (Acc+) Negative accuracy (Acc-) word-level F1 LLM-judge match trigger success rate recall win rate

Baselines: GPT-4o GPT-5 Gemini-2.0-Flash Gemini-3.0 Pro Qwen3-VL-8B Qwen3-VL-30B-A3B RAP RePIC

Builds on: GSPO RAP RePIC VLM-R1 LoRA

Closest papers by meaning

• MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.90)
• MMPB: It’s Time for Multi-Modal Personalization (NeurIPS 2025 · similarity 0.89)
• Ego: Embedding-Guided Personalization of Vision-Language Models (CVPR 2026 · similarity 0.89)
• Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.88)
• RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models (NeurIPS 2025 · similarity 0.88)
• Personalized Visual Instruction Tuning (ICLR 2025 · similarity 0.87)