Research map / Vision-language and multimodal / Human and embodied modeling

Yo'Chameleon: Personalized Vision and Language Generation

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Language-grounded visual concept learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Yo'Chameleon personalizes the unified multimodal model Chameleon for both question answering and image generation about a concept from 3-5 images, using frozen-model soft prompts with soft-positive hard negatives and self-prompting between dual token sets.

Problem

Large multimodal models are generic and lack personalized knowledge of user concepts; personalizing them for image generation via fine-tuning causes catastrophic forgetting, while soft prompts with only 3-5 images yield poor generation quality and joint training degrades both tasks.

Approach

Represent a concept as two sets of learnable soft tokens (generation and understanding) trained with a self-prompting step where the model first predicts which token set to use, and treat visually similar retrieved hard negatives as "soft positives" with adaptive token lengths proportional to their similarity to true positives.

Key results

• Yo'Chameleon achieves CLIP-I of 0.783 for personalized image generation versus 0.566 for Chameleon with image prompts and 0.657 for GPT-4o with image prompts, using 32 tokens.
• Recognition accuracy rises from 0.727 (Chameleon with text prompts) to 0.845 with 32 learnable tokens.
• Self-prompting with separate token sets reaches recognition 0.747 and CLIP-I 0.761, outperforming concatenation (0.502 / 0.615) and joint mixture training (0.564 / 0.687).

Measured effects reported in the paper

Surprising findings

• Fine-tuning for image generation causes rapid loss of the base model's world knowledge, while jointly training one soft prompt on both tasks degrades both tasks.
• Adding image prompts to Chameleon often hurts recognition accuracy and QA performance compared with no image prompt.

Limitations

• Relies on the less capable Chameleon base model, which lags GPT-4o on question answering.
• Facial similarity remains low (0.212) below the stated good-match threshold of 0.4, so human-face generation is still imprecise.
• Self-prompting token design is only examined for text and image tasks and is left for other modalities as future work.

Experimental setup

Datasets: Yo'LLaVA dataset LAION-5B

Benchmarks: Yo'LLaVA recognition and QA evaluation Personalized image generation (CLIP-I, facial similarity)

Metrics: weighted recognition accuracy QA accuracy CLIP image similarity ArcFace facial similarity

Baselines: Chameleon with text prompting Chameleon with image prompting GPT-4o with text prompting GPT-4o with image prompting Positive-only soft prompt Data augmentation with inpainting Shared learnable prompt

Builds on: Chameleon Yo'LLaVA prompt tuning SuTI Anole

Closest papers by meaning

• MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.91)
• Yo'LLaVA: Your Personalized Language and Vision Assistant (NeurIPS 2024 · similarity 0.90)
• RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
• Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting (CVPR 2024 · similarity 0.88)
• Ego: Embedding-Guided Personalization of Vision-Language Models (CVPR 2026 · similarity 0.88)
• Unified Personalized Understanding, Generating and Editing (CVPR 2026 · similarity 0.88)