Research map / Image generation / Text-to-image diffusion
PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PortraitBooth, a tuning-free one-shot text-to-portrait model that injects TFace face-recognition embeddings into Stable Diffusion's text conditioning, with identity loss and emotion-aware cross-attention control for expression editing.
Problem
Subject-driven personalized image generation either needs costly test-time fine-tuning or encoder-based methods suffer identity distortion and poor expression editability.
Approach
Augment the identity token's text embedding with features from a pre-trained face recognition model (shallow TFace features, concatenated and projected by an MLP), add a dynamic identity loss computed on one-step-estimated x0 faces at small timesteps, and apply truncated cross-attention control over face regions so identity and emotion tokens can share the face area.
Key results
- Single-subject identity preservation 0.657 vs 0.605 for Subject-Diffusion and 0.514 for FastComposer, with ~2s test time and 3 A100 training.
- Multi-subject identity preservation 0.647 vs 0.431 for FastComposer, though CLIP-TI (0.239) is lower than FastComposer (0.243) and Textual Inversion-level consistency.
- Expression coefficient 0.193 vs 0.182 for Custom Diffusion and 0.133 for FastComposer; removing STEA drops identity preservation from 0.657 to 0.563.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PortraitBooth instead of Subject-Diffusion | Single-subject personalization · Single-subject image generation | Id Pres.: 0.605 -> 0.657 | Helped |
| Use PortraitBooth instead of FastComposer | Single-subject personalization · Single-subject image generation | CLIP-TI: 0.243 -> 0.245 | Helped |
| Use PortraitBooth instead of FastComposer | Multi-subject generation · Multi-subject image generation | Id Pres.: 0.431 -> 0.647 | Helped |
| Use PortraitBooth instead of Custom Diffusion | Multi-subject generation · Multi-subject image generation | CLIP-TI: 0.258 -> 0.239 | Hurt |
| Add expression-aware cross-attention control (ablation) | PortraitBooth · Facial expression editing | Expression Coefficients: 0.177 -> 0.193 | Helped |
| Add STEA using face-recognition features rather than CLIP image-encoder features (ablation) | PortraitBooth · Single-subject image generation | Id Pres.: 0.563 -> 0.657 | Helped |
| Add dynamic identity preservation (ablation) | PortraitBooth · Single-subject image generation | Id Pres.: 0.638 -> 0.657 | Helped |
| Add emotion-aware cross-attention control (ablation) | PortraitBooth · Single-subject image generation | Id Pres.: 0.632 -> 0.657 | Helped |
| Use specific {“man”,“woman”} identity tokens instead of {“person”} (ablation) | PortraitBooth · Single-subject image generation | Id Pres.: 0.623 -> 0.657 | Helped |
| Use specific {“man”,“woman”} identity tokens instead of {“he”,“she”} (ablation) | PortraitBooth · Single-subject image generation | Id Pres.: 0.606 -> 0.657 | Helped |
| Use face mask instead of person mask (ablation) | PortraitBooth · Single-subject image generation | Id Pres.: 0.623 -> 0.657 | Helped |
| Set γ to 0.1 instead of 0.2 with β = 0.8 (ablation) | PortraitBooth · Single-subject image generation | CLIP-TI: 0.223 -> 0.245 | Helped |
Surprising findings
- Multi-subject generation takes about 18s versus ~2s for single-subject, due to the MultiDiffusion-based inference.
- Using a CLIP image encoder instead of the face recognition model for STEA sharply reduces identity preservation (0.563).
Limitations
- Multi-subject generation is slow (~18s) and relies on MultiDiffusion, since the model was trained only on single-subject data.
- Prompt consistency for multi-subject generation is weaker than FastComposer and Custom Diffusion.
Experimental setup
Datasets: CelebV-Text
Benchmarks: single-subject personalization test set (15 subjects, 30 prompts) multi-subject test set (105 pairs, 21 prompts each) expression editing test set (11 prompts per subject)
Metrics: identity preservation (FaceNet similarity) CLIP text-image consistency expression coefficient test time training cost
Baselines: Textual Inversion DreamBooth Custom Diffusion FastComposer Subject-Diffusion Face0 Stable Diffusion
Builds on: Stable Diffusion FastComposer Prompt-to-Prompt MultiDiffusion
Closest papers by meaning
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.88)
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.88)
- HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models (CVPR 2024 · similarity 0.87)
- DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation (CVPR 2024 · similarity 0.87)
- MagiCapture: High-Resolution Multi-Concept Portrait Customization (AAAI 2024 · similarity 0.87)
- FaithfulFaces: Pose-Faithful Facial Identity Preservation for Text-to-Video Generation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.