Research map / Vision-language and multimodal / Multimodal LLMs

Ego: Embedding-Guided Personalization of Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Ego, a training-free LVLM personalization method that extracts the most attended visual tokens (via keyword cross-attention in automatically selected layers) as in-context concept memories, evaluated across recognition, VQA, captioning, and video settings.

Problem

Personalizing large vision-language models to recognize and reason about user-specific concepts without per-concept fine-tuning, extra training stages, or external vision modules, and with consistent evaluation across single-concept, multi-concept, and video settings.

Approach

Use the LVLM's own cross-attention from generated keyword tokens to visual tokens to select a small, size-adaptive subset of reference-image tokens (layers chosen automatically via COCO segmentation overlap) as a concept memory injected as soft prompts in context at inference, with no training or external modules.

Key results

• On the This-is-my multi-concept split, Ego reaches 88.6 F1 in recognition (1 view) and 72.2 VQA accuracy, exceeding RAP (53.7 VQA) by nearly 20 points.
• Ego with 5 reference views reaches 85.7 F1 on Yo'LLaVA recognition, above PeKit's 82.1 F1 (single-concept, InternVL3-14B), while using a training-free pipeline.
• Ego improves captioning recall on MyVLM by about 14% over R2P and about 30% over RAP on the multi-concept This-is-my split.

Surprising findings

• Full-caption cross-attention performs slightly worse than keyword-based attention, and sampling multiple keyword sets degrades results.
• Full reference views in-context reach perfect multi-concept precision but much lower recall, so attention-based token selection outperforms raw reference images.

Limitations

• Relies on the LVLM having strong in-context visual reasoning; smaller models (2B) show reduced precision.
• Performance depends on an instruction prompt tailored to each base model; Qwen2.5-VL required a different recognition prompt.
• Number of in-context concepts is limited by the LVLM context window; no scaling evaluation beyond the reported datasets.

Experimental setup

Datasets: MyVLM Yo'LLaVA This-is-my RAP COCO 2017 train (layer calibration)

Benchmarks: MyVLM Yo'LLaVA This-is-my RAP

Metrics: Precision Recall F1-score VQA accuracy captioning recall

Baselines: RAP R2P PeKit MyVLM Yo'LLaVA Full Visual Uniform

Builds on: RAP R2P PeKit MyVLM Yo'LLaVA

Closest papers by meaning

• MMPB: It’s Time for Multi-Modal Personalization (NeurIPS 2025 · similarity 0.89)
• Contextualized Visual Personalization in Vision-Language Models (ICML 2026 · similarity 0.89)
• MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.88)
• Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.88)
• Language-Instructed Vision Embeddings for Controllable and Generalizable Perception (ICLR 2026 · similarity 0.88)
• Personalized Visual Instruction Tuning (ICLR 2025 · similarity 0.88)