Research map / Image generation / Text-to-image diffusion
PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PersonaMagic, a stage-regulated single-image face personalization method that learns timestep-dependent embeddings from a lightweight MLP within a middle denoising interval and adds a Tandem Equilibrium text-encoder loss to balance attention across prompt tokens.
Problem
Single-image personalized face generation struggles to balance identity preservation (fidelity) with text-prompt editability, since learned embeddings attend to background and overwhelm other prompt tokens.
Approach
Partition the reverse diffusion process into static and dynamic stages using IoU between cross-attention maps and face masks; in the dynamic stage a lightweight MLP generates timestep-dependent embeddings conditioned on CLIP image features, and a Tandem Equilibrium loss on text-encoder self-attention rebalances attention between the learned token and other prompt tokens.
Key results
- On CelebA-HQ one-shot, PersonaMagic reaches text similarity 0.744 and identity preservation 0.358 (Table 1 ablation), versus Custom Diffusion's 0.737 and 0.283.
- Full model achieves text similarity 0.751 on LFW vs Custom Diffusion's 0.745, with identity 0.278 vs 0.179.
- Plugging PersonaMagic into PhotoMaker raises text similarity 0.775 to 0.790 and identity 0.335 to 0.352 on 25 unseen individuals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add stage regulation to Vanilla (ablation) | PersonaMagic · CelebA-HQ | Text similarity: 0.610 to 0.672 | Helped |
| Add stage regulation to Vanilla (ablation) | PersonaMagic · CelebA-HQ | Identity preservation: 0.334 to 0.363 | Helped |
| Add Tandem Equilibrium to Vanilla (ablation) | PersonaMagic · CelebA-HQ | Text similarity: 0.610 to 0.691 | Helped |
| Add Tandem Equilibrium to Vanilla (ablation) | PersonaMagic · CelebA-HQ | Identity preservation: 0.334 to 0.284 | Hurt |
| Add stage regulation and Tandem Equilibrium (ablation) | PersonaMagic · CelebA-HQ | Text similarity: 0.610 to 0.744 | Helped |
| Add stage regulation and Tandem Equilibrium (ablation) | PersonaMagic · LFW | Identity preservation: 0.226 to 0.278 | Helped |
| Set stage thresholds to λ1=0.7 and λ2=0.8 (ablation) | PersonaMagic · CelebA-HQ | Identity preservation: 0.284 to 0.358 | Helped |
| Increase λ1 from 0.5 to 0.8 with λ2=1.0 (ablation) | PersonaMagic · CelebA-HQ | Text similarity: 0.691 to 0.749 | Helped |
| Increase λ1 to 0.8 with λ2=1.0 (ablation) | PersonaMagic · CelebA-HQ | Identity preservation: 0.328 to 0.302 | Hurt |
| Add PersonaMagic to PhotoMaker | PhotoMaker · Collected non-celebrities | Text similarity: 0.775 to 0.790 | Helped |
| Add PersonaMagic to PhotoMaker | PhotoMaker · Collected non-celebrities | Identity preservation: 0.335 to 0.352 | Helped |
| Add PersonaMagic to IP-Adapter | IP-Adapter · Collected non-celebrities | Text similarity: 0.762 to 0.778 | Helped |
Surprising findings
- Adding an identity loss to Custom Diffusion did not improve identity metrics and slightly decreased them (0.283 to 0.270), attributed to blurred X0 predictions at large timesteps.
- DreamBooth achieves strong identity preservation but lower text similarity, and Textual Inversion and Perfusion have negative identity scores on CelebA-HQ due to no-face generations.
Limitations
- Struggles to combine multiple concepts (e.g., a person and a cat) because cross-attention query weights are not fine-tuned.
- Relies on fixed thresholds lambda1=0.7 and lambda2=0.8 selected via IoU analysis on CelebA-HQ.
Experimental setup
Datasets: CelebA-HQ FFHQ LFW
Benchmarks: CelebA-HQ FFHQ LFW
Metrics: CLIPScore text similarity identity similarity (CosFace, MTCNN-detected faces) 2AFC user preference
Baselines: Textual Inversion DreamBooth Custom Diffusion NeTI Perfusion PhotoMaker IP-Adapter
Builds on: Textual Inversion NeTI DreamBooth Stable Diffusion
Closest papers by meaning
- Visual Persona: Foundation Model for Full-Body Human Customization (CVPR 2025 · similarity 0.88)
- DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation (CVPR 2024 · similarity 0.88)
- PersonaHOI: Effortlessly Improving Face Personalization in Human-Object Interaction Generation (CVPR 2025 · similarity 0.87)
- DreamIdentity: Enhanced Editability for Efficient Face-Identity Preserved Image Generation (AAAI 2024 · similarity 0.87)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.87)
- Foundation Cures Personalization: Improving Personalized Models’ Prompt Consistency via Hidden Foundation Knowledge (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.