Research map / Image generation / Editing and style transfer
Data Synthesis with Diverse Styles for Face Recognition via 3DMM-Guided Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MorphFace, a latent diffusion face generator conditioned on FR identity embeddings and DECA 3DMM renderings, with subject-aware style sampling and frequency-aware context blending, to synthesize identity-consistent, style-diverse training data for face recognition.
Problem
Identity-preserving face synthesis for training face recognition models faces a trade-off between intra-class identity consistency and style variation; prior methods treat style as subject-agnostic and offer coarse style control.
Approach
Use parametric DECA 3DMM renderings as style context alongside FR identity embeddings in an LDM, sample style attributes from class-wise (subject-aware) Gaussian distributions to model subject distinctiveness, and blend style/identity classifier-free guidance across denoising timesteps (style early, identity late).
Key results
- MorphFace 0.5M synthetic dataset averages 93.32% FR accuracy across five benchmarks, outperforming the best prior SOTA (CemiFace 92.30%).
- At 1.2M images, MorphFace averages 94.17%, above Arc2Face (93.15%) and the real CASIA baseline (94.79% average is higher, but MorphFace matches CASIA on CPLFW/CALFW).
- Context blending raises average FR accuracy from 93.11% (without blending) to 93.32% and improves eIR and identity cosine similarity relative to no blending.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use subject-aware style sampling rather than uncontrolled style generation (ablation) | MorphFace · FR benchmarks average | accuracy: 91.59 -> 93.32 | Helped |
| reuse the reference image's style rather than sample novel styles (ablation) | MorphFace · FR benchmarks average | accuracy: 82.60 -> 93.32 | Hurt |
| sample styles uniformly from the global distribution rather than subject-aware distributions (ablation) | MorphFace · FR benchmarks average | accuracy: 92.41 -> 93.32 | Hurt |
| add subject-aware style sampling (ablation) | MorphFace · FR benchmarks average | accuracy: 92.41 -> 93.32 | Helped |
| omit context blending and CFG adjustment during denoising (ablation) | MorphFace · FR benchmarks average | accuracy: 93.11 -> 93.32 | Hurt |
| apply CFG only to the identity context instead of blending identity and style guidance (ablation) | MorphFace · FR benchmarks average | accuracy: 92.75 -> 93.32 | Hurt |
| apply CFG only to the style context instead of blending identity and style guidance (ablation) | MorphFace · FR benchmarks average | accuracy: 92.83 -> 93.32 | Hurt |
| use real-world CASIA style attributes instead of statistically sampled attributes (ablation) | MorphFace · CFP-FP | accuracy: 94.11 -> 94.79 | Helped |
| use VAE-learned style attributes instead of statistically sampled attributes (ablation) | MorphFace · CFP-FP | accuracy: 91.61 -> 94.11 | Hurt |
| train on 0.5M MorphFace images rather than the strongest baseline dataset for each benchmark | MorphFace · CFP-FP | accuracy: +2.24 | Helped |
| train on 0.5M MorphFace images rather than the strongest baseline dataset for each benchmark | MorphFace · CPLFW | accuracy: +1.08 | Helped |
| train on 0.5M MorphFace images rather than the strongest baseline dataset for each benchmark | MorphFace · FR benchmarks average | accuracy: +1.02 | Helped |
Surprising findings
- Replicating the reference image's style yields much lower style variation (eIR 0.178), and uniform style sampling degrades identity consistency, showing a trade-off between intra-class consistency and variation.
- Model-learned style attributes (VAE) perform worse than statistically sampled ones, as they fail to capture subject distinctiveness.
Limitations
- Trade-off between identity consistency and style variation is only mitigated, not eliminated; style control may impair identity retention at high variation.
- Real-world CASIA style attributes outperform sampled styles, indicating room for improvement in the sampling prior.
- Frequency analysis is acknowledged as suggestive rather than definitive.
Experimental setup
Datasets: CASIA-WebFace LFW CFP-FP AgeDB CPLFW CALFW
Benchmarks: LFW CFP-FP AgeDB CPLFW CALFW
Metrics: face recognition accuracy extended Improved Recall (eIR) average pairwise cosine similarity LPIPS FFT frequency variance
Baselines: SynFace SFace DigiFace IDnet DCFace IDiff-Face ExFaceGAN SFace2 Arc2Face ID3 CemiFace
Builds on: latent diffusion models IDiff-Face DCFace DECA classifier-free guidance
Closest papers by meaning
- $\text{ID}^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition (NeurIPS 2024 · similarity 0.90)
- HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere (ICLR 2025 · similarity 0.88)
- UIFace: Unleashing Inherent Model Capabilities to Enhance Intra-Class Diversity in Synthetic Face Recognition (ICLR 2025 · similarity 0.87)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.87)
- Vec2Face: Scaling Face Dataset Generation with Loosely Constrained Vectors (ICLR 2025 · similarity 0.87)
- CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.