Research map / Image generation / Text-to-image diffusion
When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a W+ adapter that maps StyleGAN W+ face latents into Stable Diffusion v1.5 via a mapping network and residual cross-attention, with two training stages and regularized losses, enabling identity-preserving personalized generation with StyleGAN-style attribute editing.
Problem
Personalized text-to-image generation from a single reference face suffers from entangled identity and attribute information in text embedding space, causing either copied facial attributes or weak identity preservation, and limited control over attributes.
Approach
Invert the reference identity into StyleGAN's extended W+ latent space (via e4e) and align it with Stable Diffusion through a mapping network and residual cross-attention, plus losses (disentanglement, regularization) that confine W+ edits to the face region so StyleGAN edit directions transfer to diffusion outputs.
Key results
- Achieves CLIP Score .267 and ID .418 on the 120-identity CelebA-HQ comparison, on par with FastComposer (.265/.419) and above IP-Adapter-Face in CLIP score (.241) with ID .407.
- In Stage I inversion, ID .435 / FID 31.47 / LPIPS .263 versus e4e .431 / 32.26 / .205, comparable to e4e.
- Editing ID under smile and age edits is lower than e4e (.275 vs .312 for smile; .393 vs .409 for age).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the W+ adapter for personalized generation | Stable Diffusion v1.5 · Personalized generation evaluation | CLIP Score: .241 -> .267 | Helped |
| use the W+ adapter for personalized generation | Stable Diffusion v1.5 · Personalized generation evaluation | ID: .407 -> .418 | Hurt |
| use the W+ adapter for personalized generation | Stable Diffusion v1.5 · Personalized generation evaluation | Detection: .958 -> .950 | No clear effect |
| add the W+ alignment to the e4e inversion setup | Face inversion · Face inversion | ID: .431 -> .435 | Hurt |
| add the W+ alignment to the e4e inversion setup | Face inversion · Face inversion | FID: 32.26 -> 31.47 | Helped |
| add the W+ alignment to the e4e inversion setup | Face inversion · Face inversion | LPIPS: .205 -> .263 | Hurt |
| add W+ alignment and semantic attribute editing | Face attribute editing (Age) · Age edit | ID: .409 -> .393 | Helped |
| add W+ alignment and semantic attribute editing | Face attribute editing (Smile) · Smile edit | ID: .312 -> .275 | Helped |
| replace residual cross-attention with parallel cross-attention (ablation) | W+ adapter · Personalized generation evaluation | CLIP Score: .267 -> .243 | Hurt |
| replace residual cross-attention with parallel cross-attention (ablation) | W+ adapter · Personalized generation evaluation | ID: .516 -> .526 | Hurt |
| remove the regularization loss Lreg (ablation) | W+ adapter · Personalized generation evaluation | CLIP Score: .267 -> .136 | Hurt |
| remove the regularization loss Lreg (ablation) | W+ adapter · Personalized generation evaluation | ID: .516 -> .461 | Helped |
Surprising findings
- IP-Adapter tends to ignore text conditions in some cases despite strong identity retention.
- Removing the regularization loss raises identity-related scores but drops CLIP score substantially (.136 vs .267), showing an identity-text trade-off.
Limitations
- Inversion through e4e loses identity detail, limiting identity fidelity.
- Struggles to edit some attributes such as pose and glasses.
- Supports only a single face per generation.
Experimental setup
Datasets: FFHQ StyleGAN2 synthetic faces SHHQ CelebA-HQ
Benchmarks: CelebA-HQ (120 identities, 20 prompts) CelebA-HQ (1,000 images for inversion/editing)
Metrics: CLIP Score ID (ArcFace distance) Face Detection Score (MTCNN) FID LPIPS
Baselines: Textual Inversion Dreambooth Custom Diffusion FastComposer IP-Adapter-Face CelebBasis
Builds on: Stable Diffusion IP-Adapter e4e InterFaceGAN BLIP-2
Closest papers by meaning
- DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation (CVPR 2024 · similarity 0.89)
- Cross Initialization for Face Personalization of Text-to-Image Models (CVPR 2024 · similarity 0.89)
- PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization (CVPR 2024 · similarity 0.88)
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network (AAAI 2025 · similarity 0.88)
- OSDFace: One-Step Diffusion Model for Face Restoration (CVPR 2025 · similarity 0.88)
- DreamIdentity: Enhanced Editability for Efficient Face-Identity Preserved Image Generation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.