Research map / Image generation / Text-to-image diffusion
Cross Initialization for Face Personalization of Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cross Initialization, which initializes Textual Inversion's face embedding with the CLIP text encoder's output of a mean name embedding, narrowing the initial-learned gap and cutting optimization to 320 steps for face personalization on Stable Diffusion v2.1.
Problem
Textual Inversion-based face personalization overfits the input image, trading off reconstruction quality against editability; the learned embedding drifts far (up to 100x in scale) from its coarse super-category initialization.
Approach
Initialize the textual embedding with the text encoder's output E(mean embedding of 691 well-known names) so the initial embedding is close to the learned one in scale and orientation, combined with an L2 regularization toward this initialization.
Key results
- Ours achieves identity similarity 0.2517 and prompt similarity 0.2859 versus Textual Inversion's 0.2115 and 0.2498, with personalization time of 346 s vs 6331 s.
- Ours-fast personalizes in 25 steps (about 26 s) with identity 0.2225 and prompt 0.2800, surpassing Textual Inversion and Celeb Basis on prompt similarity.
- In a user study with 600 responses from 30 participants, users preferred Ours over Textual Inversion (78.0%), DreamBooth (90.7%), NeTI (75.3%), and Celeb Basis (73.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Cross Initialization (ablation) | Textual Inversion for face personalization · CelebA-HQ test set | identity similarity | Helped |
| Add a mean textual embedding of 691 well-known names (ablation) | Cross Initialization · CelebA-HQ test set | prompt alignment | Helped |
| Add the regularization term that keeps the learned embedding close to its initialization (ablation) | Cross Initialization · CelebA-HQ test set | prompt alignment | Helped |
| Use Ours-fast instead of Celeb Basis | Face personalization on CelebA-HQ · CelebA-HQ test set | identity similarity: 0.2070 -> 0.2225 | Helped |
| Use Ours-fast instead of Celeb Basis | Face personalization on CelebA-HQ · CelebA-HQ test set | prompt similarity: 0.2683 -> 0.2800 | Helped |
| Use Ours-fast instead of Textual Inversion | Face personalization on CelebA-HQ · CelebA-HQ test set | personalization time (s): 6331 -> 26 | Helped |
| Use Cross Initialization instead of traditional Textual Inversion initialization | Face personalization · Face expression editing | expression editability | Helped |
Surprising findings
- Using the text encoder output as the initialization yields an image nearly identical to the one from the raw embedding, and the learned embedding's scale can reach 100x the initial embedding's scale in Textual Inversion.
- NeTI achieves the highest identity similarity (0.3789) but the lowest prompt similarity (0.2325), indicating strong overfitting.
Limitations
- For general (non-human) concepts, Cross Initialization is reported as not as effective as for human faces.
- Evaluation uses a single-image setting on 200 CelebA-HQ images and 20 prompts; the identity metric excludes stylization prompts since ArcFace is trained on real images.
Experimental setup
Datasets: CelebA-HQ 691 well-known names (for mean textual embedding)
Benchmarks: CelebA-HQ test set (200 images, 20 prompts)
Metrics: identity similarity (ArcFace) prompt similarity (CLIP score) personalization time user study preference
Baselines: Textual Inversion DreamBooth NeTI Celeb Basis
Builds on: Textual Inversion Stable Diffusion Celeb Basis
Closest papers by meaning
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.89)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- Face2Diffusion for Fast and Editable Face Personalization (CVPR 2024 · similarity 0.87)
- Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.86)
- Efficient Personalization of Quantized Diffusion Model without Backpropagation (CVPR 2025 · similarity 0.86)
- PFStorer: Personalized Face Restoration and Super-Resolution (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.