Research map / Image generation / Text-to-image diffusion
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
PhotoMaker encodes multiple ID images into a stacked ID embedding that replaces the class word in SDXL's text embedding, enabling tuning-free personalized human photo generation trained on an ID-grouped web dataset.
Problem
Personalized text-to-image generation of human photos currently trades off between ID fidelity, efficiency (tuning-free), and text controllability; DreamBooth-style tuning is slow and single-image embedding methods lose fidelity and editability.
Approach
Stack CLIP image embeddings of multiple same-ID images (fused with the class-word text feature via MLPs) and substitute them for the class word in the text embedding, so SDXL's own cross-attention integrates ID and prompt; training uses an automatically built ID-grouped human dataset, and at inference the stack can mix different IDs.
Key results
- PhotoMaker reaches CLIP-T 26.1, CLIP-I 73.6, DINO 51.5 and face diversity 57.7 on the recontextualization benchmark, with about 10 s per personalization on a V100 (vs. 1284 s for DreamBooth).
- Stacked ID embedding beats averaging and linear-projection composition in DINO (49.5 vs 47.0/47.3) and face similarity (53.6 vs 48.8/48.1) in ablation.
- Method is reported about 130x faster than DreamBooth when using four ID images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DreamBooth with PhotoMaker | SDXL personalized generation · Universal recontextualization | DINO: 39.8 -> 51.5 | Helped |
| replace IPAdapter with PhotoMaker | SDXL personalized generation · Universal recontextualization | CLIP-I: 71.2 -> 73.6 | Helped |
| replace IPAdapter with PhotoMaker | SDXL personalized generation · Universal recontextualization | Face Similarity: 67.1 -> 61.8 | Hurt |
| replace Textual Inversion with PhotoMaker | SDXL personalized generation · Universal recontextualization | FID: 363.5 -> 370.3 | Hurt |
| use stacked ID embedding instead of average embeddings (ablation) | PhotoMaker · Embedding composing choices | DINO: 47.0 -> 49.5 | Helped |
| use stacked ID embedding instead of linear projection (ablation) | PhotoMaker · Embedding composing choices | Face Similarity: 48.1 -> 53.6 | Helped |
| use stacked ID embedding instead of average embeddings (ablation) | PhotoMaker · Embedding composing choices | CLIP-T: 28.7 -> 28.0 | Hurt |
| use stacked ID embedding instead of average embeddings (ablation) | PhotoMaker · Embedding composing choices | Face Diversity: 56.3 -> 55.0 | Hurt |
| use proposed data instead of FFHQ-wild (ablation) | PhotoMaker trained with a single embedding · Data and embedding contribution | CLIP-T: 27.1 -> 27.0 | No clear effect |
| use proposed data with single embedding instead of FFHQ-wild (ablation) | PhotoMaker · Data and embedding contribution | Face Similarity: 62.0 -> 63.5 | Helped |
| add stacked ID embedding to the proposed data training (ablation) | PhotoMaker · Data and embedding contribution | CLIP-T: 27.0 -> 28.6 | Helped |
| add stacked ID embedding to the proposed data training (ablation) | PhotoMaker · Data and embedding contribution | DINO: 49.1 -> 45.3 | Hurt |
Surprising findings
- Using the authors' dataset with a single embedding gave no significant improvement over the FFHQ-wild baseline; the stacked embedding was the main source of gains in face diversity and text consistency.
Limitations
- Training data built from celebrity web images, which may limit generalization to non-celebrity identities and raises ethical concerns about identity use.
- Identity fidelity is slightly below IP-Adapter on face similarity (61.8 vs 67.1) and FastComposer-level in some metrics.
Experimental setup
Datasets: VGGFace2 (name roster) FFHQ-wild MyStyle (evaluation) ID-oriented dataset built by the authors
Benchmarks: universal recontextualization evaluation (25 IDs, 40 prompts) MyStyle IDs subset
Metrics: CLIP-T CLIP-I DINO Face Similarity Face Diversity (LPIPS) FID speed
Baselines: DreamBooth Textual Inversion FastComposer IPAdapter SDXL
Builds on: SDXL DreamBooth FastComposer IP-Adapter CLIP
Closest papers by meaning
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.87)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.86)
- ID-Patch: Robust ID Association for Group Photo Personalization (CVPR 2025 · similarity 0.85)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.85)
- Customization Assistant for Text-to-Image Generation (CVPR 2024 · similarity 0.85)
- CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.