Research map / Image generation / Editing and style transfer
Face2Diffusion for Fast and Editable Face Personalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Face2Diffusion, a tuning-free face personalization method that removes identity-irrelevant information (camera pose, expression, background) from the encoder and training objective of Stable Diffusion to improve editability.
Problem
Face personalization inserts specific faces into text-to-image diffusion models, but existing methods overfit to training samples, losing editability (text alignment of backgrounds, poses, expressions) while preserving identity.
Approach
Three components remove identity-irrelevant information: a multi-scale identity encoder trained with ArcFace-style multi-scale losses to disentangle pose from identity, expression guidance that feeds 3D-reconstruction expression features with random dropout to a learnable unconditional vector, and class-guided denoising regularization that forces background denoising to match the class prompt "a p
Key results
- Face2Diffusion achieves the best Identity×Text hMean (0.2252) and gMean (0.2252 top-1 ranking) among compared methods, ranking top-3 in five of six metrics.
- MSID encoder raises shallow-layer ArcFace AUC from 93.99% to 99.46% and improves CLIP editability from 0.2069 to 0.2486 versus multi-scale ArcFace features.
- Face2Diffusion has 1.20e7 trainable parameters and encoding time of 0.006 sec, the fastest among compared methods, with best FID of 69.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the MSID encoder instead of ArcFace multi-scale features (ablation) | Face2Diffusion · FaceForensics++ | CLIP: 0.2069 to 0.2486 | Helped |
| use the MSID encoder instead of the original ArcFace (ablation) | Face2Diffusion · FaceForensics++ | AdaFace: 0.2421 to 0.3143 | Helped |
| add expression guidance (ablation) | Face2Diffusion · FaceForensics++ | CLIP: 0.2315 to 0.2486 | Helped |
| add expression guidance (ablation) | Face2Diffusion · FaceForensics++ | AdaFace: 0.3338 to 0.3143 | Hurt |
| replace original reconstruction training with CGDR (ablation) | Face2Diffusion · FaceForensics++ | CLIP: 0.2053 to 0.2486 | Helped |
| replace original reconstruction training with CGDR (ablation) | Face2Diffusion · FaceForensics++ | hMean: 0.1411 to 0.2252 | Helped |
| add delayed subject conditioning to reconstruction training (ablation) | Face2Diffusion · FaceForensics++ | CLIP: 0.2053 to 0.2851 | Helped |
| add delayed subject conditioning to reconstruction training (ablation) | Face2Diffusion · FaceForensics++ | AdaFace: 0.3133 to 0.1651 | Hurt |
| use masked reconstruction instead of CGDR (ablation) | Face2Diffusion · FaceForensics++ | CLIP: 0.1990 vs. 0.2486 | Hurt |
| use CGDR instead of reconstruction with DSC (ablation) | Face2Diffusion · FaceForensics++ | hMean: 0.1596 to 0.2252 | Helped |
| use Face2Diffusion instead of the strongest baseline | Face2Diffusion · FaceForensics++ | hMean: 0.1655 to 0.1749 | Helped |
| use Face2Diffusion instead of the strongest baseline | Face2Diffusion · FaceForensics++ | gMean: 0.2120 to 0.2252 | Helped |
Surprising findings
- Delayed subject conditioning (DSC) raises text-fidelity but sharply degrades identity similarity (AdaFace 0.3133 to 0.1651), showing a strong identity-text trade-off.
- CustomDiffusion gets the best identity scores but poor editability due to overfitting.
Limitations
- Identity similarity is slightly lower than without expression guidance (AdaFace 0.3143 vs 0.3338)
- Evaluation limited to 100 FaceForensics++ faces and 40 prompts
- Potential misuse for fake news generation
Experimental setup
Datasets: MS1M FFHQ VGGFace2 FaceForensics++ CC12M
Benchmarks: FaceForensics++ subset with 40 prompts
Metrics: AdaFace SphereFace FaceNet CLIP score directional CLIP SigLIP harmonic mean geometric mean FID upper-bound hMean
Baselines: TextualInversion DreamBooth CustomDiffusion Perfusion E4T CelebBasis FastComposer ELITE DreamIdentity
Builds on: Stable Diffusion ArcFace DreamIdentity Delayed subject conditioning ELITE
Closest papers by meaning
- DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation (CVPR 2024 · similarity 0.87)
- Cross Initialization for Face Personalization of Text-to-Image Models (CVPR 2024 · similarity 0.87)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.87)
- RealisID: Scale-Robust and Fine-Controllable Identity Customization via Local and Global Complementation (AAAI 2025 · similarity 0.86)
- DreamIdentity: Enhanced Editability for Efficient Face-Identity Preserved Image Generation (AAAI 2024 · similarity 0.86)
- FaceMe: Robust Blind Face Restoration with Personal Identification (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.