Research map / Large language models / Language, safety and interpretability
UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniCTokens, which personalizes a unified VLM (Show-o) with a single set of unified concept tokens trained via a three-stage progressive schedule, and introduces UnifyBench for personalized understanding, generation, and attribute-reasoning generation.
Problem
Personalized understanding and generation are treated separately, with distinct concept tokens, so information does not transfer across tasks; this limits generating images that require reasoning over user-provided attributes not present in the image training data (personalized attribute-reasoning g
Approach
Train a single set of unified concept tokens in a unified VLM with a three-stage schedule: understanding warm-up, bootstrapping generation from understanding tokens, then deepening understanding using k-means-selected hard regions found from generation intermediate-step differences.
Key results
- On UnifyBench, UniCTokens (1.3B, ~10 training images per concept) reaches PARG score 0.359 versus 0.224 for Yo'Chameleon (7B, ~1,000 images) and the best recognition weight 0.790 vs 0.764.
- On UnifyBench pure generation, UniCTokens attains CLIP-I 0.750 and CLIP-T 0.282, the best among unified models, and face similarity 0.334 vs 0.224 for Yo'Chameleon.
- On Yo'LLaVA and MC-LLaVA understanding benchmarks, UniCTokens outperforms Yo'Chameleon in recognition (0.852 vs 0.845 on Yo'LLaVA) with a 1.3B backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unified concept tokens and three-stage training instead of separate task personalization | Show-o 1.3B · UnifyBench PARG | Score: 0.770 vs. 0.359 | Mixed |
| Use unified concept tokens with three-stage training | Show-o 1.3B · UnifyBench | Rec. Weight: 0.764 -> 0.790 | Helped |
| Use unified concept tokens with three-stage training | Show-o 1.3B · UnifyBench | Pure Gen. CLIP-I: 0.697 -> 0.750 | Helped |
| Use unified concept tokens with three-stage training | Show-o 1.3B · UnifyBench | PARG Score: 0.266 -> 0.359 | Helped |
| Add Stage 1 understanding warm-up before generation training (ablation) | UniCTokens · UnifyBench PARG | Score: 0.297 -> 0.349 | Helped |
| Add generation bootstrapping after Stage 1 (ablation) | UniCTokens · UnifyBench | Pure Gen. CLIP-I: 0.524 -> 0.752 | Helped |
| Add Stage 3 generation-derived fine-grained tokens (ablation) | UniCTokens · UnifyBench | Rec. Weight: 0.621 -> 0.790 | Helped |
| Initialize Stage 3 tokens using generation-derived regions rather than no initialization (ablation) | UniCTokens · UnifyBench | Rec. Weight: 0.670 -> 0.790 | Helped |
| Initialize Stage 3 tokens using generation-derived regions rather than the full image (ablation) | UniCTokens · UnifyBench | Rec. Weight: 0.723 -> 0.790 | Helped |
| Use three-stage training instead of joint training (ablation) | UniCTokens · UnifyBench PARG | Score: 0.269 -> 0.359 | Helped |
| Increase learnable token count (ablation) | UniCTokens · Personalized generation | CLIP-T | Hurt |
| Remove Stage 2 MMU training data (ablation) | UniCTokens · Personalized generation | generative capabilities | No clear effect |
Surprising findings
- Directly training generation on shared understanding tokens (Stage 2 without the extra generation tokens) degrades understanding and loses general conditioning on the concept.
- Adding learnable tokens beyond a point (e.g., 64) reduces CLIP-T, suggesting longer contexts make conditioning harder.
Limitations
- Inherits limitations of the Show-o base model, including difficulty with varying styles.
- Does not handle domain-shift inputs well, e.g., specialized fields such as medicine.
- Human face personalization remains below the recommended facial similarity threshold (about 0.4-0.5).
Experimental setup
Datasets: UnifyBench Yo'LLaVA dataset MC-LLaVA dataset DreamBench MyVLM images
Benchmarks: UnifyBench Yo'LLaVA MC-LLaVA DreamBench GenEval MMMU POPE
Metrics: recall (weight) BLEU LLM score CLIP-I CLIP-T DINO face similarity (ArcFace) PARG score
Baselines: Yo'Chameleon Show-o+TP Chameleon+TP Chameleon+IP GPT-4o+TP GPT-4o+IP Yo'LLaVA MC-LLaVA RAP-MLLM DreamBooth Text inversion Qwen2.5-VL+TP
Builds on: Show-o Yo'Chameleon MC-LLaVA DreamBooth Text inversion
Closest papers by meaning
- Unified Personalized Understanding, Generating and Editing (CVPR 2026 · similarity 0.89)
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.88)
- OmniGen: Unified Image Generation (CVPR 2025 · similarity 0.87)
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.87)
- Restyling Unsupervised Concept Based Interpretable Networks with Generative Models (ICLR 2025 · similarity 0.87)
- Ego: Embedding-Guided Personalization of Vision-Language Models (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.