Research map / Image generation / Editing and style transfer
Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffSFSR, a latent diffusion model for simultaneous face swapping and reenactment that composes identity, expression, and background conditions to generate personalized portraits with fine-grained (135-class) emotion control.
Problem
Generating personalized face portraits that simultaneously preserve a user's identity, a specified fine-grained expression, and a text-described background, where identity and expression are highly entangled and existing expression labels are coarse (7-8 classes).
Approach
Extends latent diffusion with three conditions (masked background image, compound identity embedding from three face recognizers, and a DLN expression embedding) so swapping and reenactment are done jointly; adds explicit background conditioning during training and an improved midpoint sampling that correctly maps z_t to z_{t-1} via DDPM equations before estimating z0 for identity/expression losse
Key results
- On CelebA-HQ Swap All (SFSR), DiffSFSR reaches ID 90.2 and Exp error 0.55 versus 76.9 and 2.24 for the hybrid SimSwap+StyleHEAT method.
- In the face reenactment comparison on CelebA-HQ, DiffSFSR achieves Exp error 0.58 versus 2.43 for StyleHEAT.
- User study gives identity consistency 95.6% and expression consistency 90.4% for fine-grained expression synthesis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add explicit background conditioning (ablation) | DiffSFSR · CelebA-HQ | ID. (×100): 81.5 -> 87.0 | Helped |
| Add explicit background conditioning (ablation) | DiffSFSR · CelebA-HQ | Pose.: 8.97 -> 7.36 | Helped |
| Add explicit background conditioning (ablation) | DiffSFSR · CelebA-HQ | MSE.: 51.40 -> 51.23 | Helped |
| Add identity embedding 3 (ablation) | DiffSFSR · CelebA-HQ | ID. (×100): 74.9 -> 87.0 | Helped |
| Add identity embedding 2 (ablation) | DiffSFSR · CelebA-HQ | ID. (×100): 37.9 -> 87.0 | Helped |
| Use DiffSFSR instead of the hybrid method | simultaneous face swapping and reenactment · CelebA-HQ | ID. (×100): 76.9 -> 90.2 | Helped |
| Use DiffSFSR instead of the hybrid method | simultaneous face swapping and reenactment · CelebA-HQ | Exp. (×100): 2.24 -> 0.55 | Helped |
| Use DiffSFSR instead of the hybrid method | simultaneous face swapping and reenactment · FF++ | Pose.: 5.67 -> 5.19 | Helped |
| Use DiffSFSR instead of StyleHEAT | face reenactment · CelebA-HQ | Exp. (×100): 2.43 -> 0.58 | Helped |
Surprising findings
- The hybrid baseline of combining the best swapping and reenactment methods performs far worse than the joint model on the simultaneous task, showing a single model is needed.
Limitations
- Identity-free expression encoding remains an open problem, so residual identity leaks through the expression embedding and needs compensation via compound identity embedding.
- Pose error for SFSR (6.00 on CelebA-HQ) is worse than the best swap-only method, indicating a trade-off.
Experimental setup
Datasets: CelebA-HQ FFHQ emotion dataset of 728,946 images across 135 categories
Benchmarks: CelebA-HQ FF++
Metrics: ID retrieval accuracy expression error pose error MSE user study scores
Baselines: SDXL SimSwap FaceShifter HifiFace E4S DiffSwap Face2Face StyleHEAT
Builds on: latent diffusion model DiffSwap DDPM DLN SDXL
Closest papers by meaning
- PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization (CVPR 2024 · similarity 0.84)
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network (AAAI 2025 · similarity 0.83)
- Foundation Cures Personalization: Improving Personalized Models’ Prompt Consistency via Hidden Foundation Knowledge (NeurIPS 2025 · similarity 0.83)
- PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium (AAAI 2025 · similarity 0.82)
- RealisID: Scale-Robust and Fine-Controllable Identity Customization via Local and Global Complementation (AAAI 2025 · similarity 0.81)
- PersonaHOI: Effortlessly Improving Face Personalization in Human-Object Interaction Generation (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.