Research map / Image generation / Text-to-image diffusion
DreamIdentity: Enhanced Editability for Efficient Face-Identity Preserved Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamIdentity, an optimization-free face-identity encoder (M^2 ID encoder with multi-scale ViT features and multi-word projection) plus self-augmented editability learning from celebrity images generated by Stable Diffusion, enabling identity-preserving text-to-image generation in 0.04 s per face.
Problem
Preserving a conditioned face's identity while following text prompts in text-to-image generation, without per-identity optimization and without harming the editability of the pre-trained T2I model.
Approach
Replace the CLIP last-layer feature with a face-recognition ViT encoder whose multi-layer CLS features are projected to multiple pseudo-word embeddings, and train the encoder with a self-generated celebrity editing dataset (generated by the T2I model itself) so that editing ability transfers to unseen faces, while the T2I model's weights stay frozen.
Key results
- Ours reaches text-alignment 0.228 and face similarity 0.467, versus 0.220/0.420 for E4T and 0.196/0.450 for ELITE.
- Encoding takes 0.04 s per face, versus 20 min for Textual Inversion, 4 min for DreamBooth and 20 s for E4T.
- Replacing CLIP with the face encoder raises face similarity from 0.266 to 0.302; adding multi-scale features gives 0.412 and two embeddings gives 0.467.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the CLIP encoder with a face-specific ID encoder (ablation) | DreamIdentity · Face identity re-contextualization test set | Face-similarity: 0.266 to 0.302 | Helped |
| Add multi-scale features to the face-specific ID encoder (ablation) | DreamIdentity · Face identity re-contextualization test set | Face-similarity: 0.302 to 0.412 | Helped |
| Add multi-word embeddings (2 embeddings) (ablation) | DreamIdentity with multi-scale ID features · Face identity re-contextualization test set | Face-similarity: +12% | Helped |
| Add multi-word embeddings (2 embeddings) (ablation) | DreamIdentity with multi-scale ID features · Face identity re-contextualization test set | Text-alignment: 0.4% | No clear effect |
| Increase the number of word embeddings beyond 2 (ablation) | DreamIdentity · Face identity re-contextualization test set | Text-alignment: 17% | Hurt |
| Train with self-augmented editability learning instead of reconstruction objective alone (ablation) | DreamIdentity · Face identity re-contextualization test set | Text-alignment | Helped |
| Remove reconstruction learning on FFHQ and train only with the self-augmented editing dataset (ablation) | DreamIdentity · Face identity re-contextualization test set | Face-similarity | Hurt |
| Use DreamIdentity instead of E4T | Face identity re-contextualization test set · Face identity re-contextualization test set | Text-alignment: 7% | Helped |
Surprising findings
- Increasing the number of word embeddings beyond two drops text-alignment by about 17%.
Limitations
- Poor-quality or out-of-domain face inputs, such as partially occluded faces, give limited edited-image quality.
- Editability is undermined when the requested scene conflicts with the subject's gender.
Experimental setup
Datasets: FFHQ self-augmented celebrity editing dataset
Benchmarks: FFHQ test faces (100 faces)
Metrics: text-alignment (CLIP cosine) face similarity (ArcFace cosine) encoding time
Baselines: Textual Inversion DreamBooth E4T ELITE InstructPix2Pix
Builds on: Textual Inversion ELITE Latent Diffusion Models
Closest papers by meaning
- DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation (CVPR 2024 · similarity 0.91)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.88)
- PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium (AAAI 2025 · similarity 0.87)
- Face2Diffusion for Fast and Editable Face Personalization (CVPR 2024 · similarity 0.86)
- PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization (CVPR 2024 · similarity 0.86)
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.