Research map / Image generation / Text-to-image diffusion
PuLID: Pure and Lightning ID Customization via Contrastive Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PuLID, a tuning-free ID customization method for SDXL that adds a 4-step SDXL-Lightning branch to enable contrastive alignment loss (to limit disruption of the base model) and an accurate ID loss computed on generated x0.
Problem
Tuning-free identity (ID) customization for text-to-image diffusion models disrupts the base model's behavior (style, lighting, layout, prompt-following) and ID loss during training is inaccurate because predicted x0 from intermediate timesteps is noisy.
Approach
Add a Lightning T2I branch that generates images from pure noise in 4 steps with and without ID conditioning from the same prompt and latent; align UNet cross-attention features between the two paths (semantic and layout alignment losses) and compute ID loss on the resulting clean x0, matching test-time sampling.
Key results
- On DivID-120, PuLID reaches Face Sim. 0.733, CLIP-T 31.31 and CLIP-I 0.812, versus InstantID 0.725 / 28.72 / 0.680 and IPAdapter 0.619 / 28.36 / 0.703.
- On Unsplash-50, PuLID reaches Face Sim. 0.659 and CLIP-I 0.840, outperforming InstantID (0.614, 0.736) and IPAdapter (0.615, 0.701).
- Training with 4-step SDXL-Lightning is about 2.6-3.3 s/iter versus 6.6-7.3 s/iter without fast sampling; the non-accelerated variant reaches Face Sim. 0.743 on DivID-120.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PuLID instead of the strongest listed baseline | SDXL-Lightning ID customization · DivID-120 | Face Sim.: 0.725 -> 0.733 | Helped |
| Use PuLID instead of the strongest listed baseline | SDXL-Lightning ID customization · DivID-120 | CLIP-T: 28.72 -> 31.31 | Helped |
| Use PuLID instead of the strongest listed baseline | SDXL-Lightning ID customization · DivID-120 | CLIP-I: 0.703 -> 0.812 | Helped |
| Use PuLID instead of the strongest listed baseline | SDXL-Lightning ID customization · Unsplash-50 | Face Sim.: 0.615 -> 0.659 | Helped |
| Use PuLID instead of the strongest listed baseline | SDXL-Lightning ID customization · Unsplash-50 | CLIP-T: 30.55 -> 32.16 | Helped |
| Use PuLID instead of the strongest listed baseline | SDXL-Lightning ID customization · Unsplash-50 | CLIP-I: 0.736 -> 0.840 | Helped |
| Add alignment loss to the Stage2 model (ablation) | PuLID · DivID-120 | CLIP-T and CLIP-I: 24.91 -> 31.31; 0.624 -> 0.812 | Helped |
| Add alignment loss to the Stage2 model (ablation) | PuLID · Unsplash-50 | CLIP-T and CLIP-I: 25.83 -> 32.16; 0.646 -> 0.840 | Helped |
| Add naive ID loss to the Stage1 baseline (ablation) | PuLID training · DivID-120 | Face Sim.: 0.561 -> 0.652 | Helped |
| Replace naive ID loss with accurate Lightning T2I ID loss (ablation) | PuLID training · DivID-120 | Face Sim.: 0.652 -> 0.761 | Helped |
| Replace naive ID loss with accurate Lightning T2I ID loss (ablation) | PuLID training · Unsplash-50 | Face Sim.: 0.601 -> 0.708 | Helped |
| Replace fast sampling with non-accelerated SDXL training branch (ablation) | PuLID · DivID-120 | Face Sim.: 0.733 -> 0.743 | Helped |
Surprising findings
- Without the alignment loss, the ID-loss-trained model's CLIP-T and CLIP-I drop sharply (e.g., CLIP-I 0.624 at Stage 2 vs 0.812 final), indicating that maximizing ID similarity alone strongly disrupts the base model.
- Existing comparison methods (InstantID, IPAdapter) produce better results when run on SDXL-Lightning than on SDXL-base, so the authors used the accelerated base for all comparisons.
Limitations
- Training per iteration is slower and more memory-intensive than conventional diffusion training due to the Lightning branch.
- The accurate ID loss causes some blurriness in faces and reduces image quality to some extent.
- Without the accelerated base, the method is more costly; disruptions are more noticeable on base models other than SDXL-Lightning unless trained with multiple base models.
Experimental setup
Datasets: internal 1.5M human portrait dataset (Internet-collected, BLIP-2 captions) Unsplash-50
Benchmarks: DivID-120 Unsplash-50
Metrics: Face Sim. (cosine, CurricularFace) CLIP-T CLIP-I
Baselines: IPAdapter InstantID PhotoMaker
Builds on: IP-Adapter SDXL-Lightning InstantID ELITE PhotoMaker
Closest papers by meaning
- PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (CVPR 2024 · similarity 0.84)
- ID-Patch: Robust ID Association for Group Photo Personalization (CVPR 2025 · similarity 0.82)
- CustAny: Customizing Anything from A Single Example (CVPR 2025 · similarity 0.82)
- Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.82)
- DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation (ICLR 2025 · similarity 0.81)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.