Research map / Vision-language and multimodal / Visual question answering
Yo'LLaVA: Your Personalized Language and Vision Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Yo'LLaVA, which personalizes a frozen LLaVA-1.5-13B to a user-specific subject by learning a handful of latent soft-prompt tokens plus an identifier token, trained with synthetic conversations and retrieved hard negatives.
Problem
Large multimodal models answer generically and cannot recognize or converse about user-specific subjects (e.g., a user's pet dog or a friend) from only a few example images, without catastrophic forgetting or privacy-heavy large datasets.
Approach
Represent a personalized subject as a new identifier token plus k learnable latent tokens, training only those input embeddings and the identifier's output classifier weights while freezing the rest of LLaVA; generate conversation data without the reference image and mine CLIP-retrieved hard negatives from LAION for recognition.
Key results
- Yo'LLaVA with 16 learnable tokens reaches 0.924 weighted recognition accuracy vs 0.838-0.841 for GPT-4V with text prompts and 0.819-0.822 for LLaVA with ~16-token prompts.
- On the MyVLM benchmark, Yo'LLaVA attains 96.4 weighted accuracy and 100.0 recall vs MyVLM's 93.8 and 96.0.
- Catastrophic forgetting check shows POPE rand 0.86 vs 0.87 for original LLaVA, and MMBench 0.68 for both.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a learnable prompt with 16 tokens instead of a human-written text description | LLaVA · Personalized subject recognition | weighted accuracy: 0.822 -> 0.924 | Helped |
| Use a learnable prompt with 16 tokens instead of LLaVA without personalized information | LLaVA · Personalized subject recognition | weighted accuracy: 0.500 -> 0.924 | Helped |
| Use a learnable prompt with 16 tokens instead of a human-written description | LLaVA · Visual question answering | accuracy: 0.925 -> 0.929 | No clear effect |
| Use a learnable prompt with 16 tokens instead of a human-written description | LLaVA · Text-only question answering | accuracy: 0.790 -> 0.883 | Helped |
| Replace MyVLM's external recognizer with the integrated Yo'LLaVA approach | Personalized subject recognition on the MyVLM dataset · MyVLM dataset | weighted accuracy: 93.8 -> 96.4 | Helped |
| Personalize LLaVA while freezing its core weights | LLaVA-1.5-13B · MMBench en | score: 0.68 -> 0.68 | No clear effect |
| Vary the number of latent tokens from 0 to 16 (ablation) | Yo'LLaVA · Personalized subject recognition | accuracy: 24% -> 91% | Helped |
| Increase the number of training images from 1 to 5 (ablation) | Yo'LLaVA · Personalized subject recognition | accuracy | Helped |
| Add recognition training data (ablation) | Vanilla LLaVA · Personalized subject recognition | accuracy: 0.500 -> 0.707 | Helped |
| Add synthesized conversation data to recognition training (ablation) | LLaVA + Recognition · Personalized subject recognition | accuracy: 0.707 -> 0.754 | Helped |
| Add synthesized conversation data to recognition training (ablation) | LLaVA + Recognition · Text-only conversation about the subject | conversation ability | Helped |
| Add retrieved hard negative examples (ablation) | Yo'LLaVA with recognition and conversation data · Personalized subject recognition | accuracy: 0.754 -> 0.914 | Helped |
Surprising findings
- Vanilla LLaVA always answers 'No' to recognition queries, yielding 0.5 weighted accuracy.
- Lengthy text descriptions (~1.3k tokens) hurt LLaVA recognition (0.650), while short ones help.
Limitations
- Fine-grained recognition is weak when facial features are small in the image.
- Hallucinations inherited from the language model, e.g., inventing a person's birthday.
- Evaluation uses a self-collected 40-subject dataset and multiple-choice QA, limiting generality.
Experimental setup
Datasets: LAION LLaVA-generated conversations on subject images MyVLM dataset
Benchmarks: Yo'LLaVA personalization dataset recognition test visual and text QA benchmark (571 questions) MyVLM dataset POPE MMBench LLaVA-Wild
Metrics: weighted accuracy positive accuracy negative accuracy recall QA accuracy
Baselines: LLaVA LLaVA + text prompt GPT-4V MyVLM
Builds on: LLaVA prompt tuning CLIP
Closest papers by meaning
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.90)
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.87)
- RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- Ego: Embedding-Guided Personalization of Vision-Language Models (CVPR 2026 · similarity 0.86)
- MMPB: It’s Time for Multi-Modal Personalization (NeurIPS 2025 · similarity 0.86)
- Personalized Visual Instruction Tuning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.