Research map / Image generation / Text-to-image diffusion
Limitations of Face Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audits face images generated by text-to-image diffusion models (SDv2.1 and Realism) with SEGA edits, using face verification, quality metrics, and a user study, and adds an analytical GMM model explaining how biased training data yields demographic disparities.
Problem
Text-to-image diffusion models are increasingly used to generate synthetic face data for training and testing face recognition, but their faithfulness to prompts, demographic disparities, and distributional shifts are not well characterized.
Approach
Generate identity-consistent face sets by pairing demographic-specific names with fixed seeds and SEGA semantic edits (no fine-tuning or inversion), then audit them jointly with face verification, CLIP/DINO embedding metrics, and a large user study; a GMM-based analysis links biased training data to biased generation.
Key results
- Generated faces (Realism, SDv2.1) show lower face verification accuracy than natural LFW faces across demographics, except Realism Celebrities for the White group.
- User study: SEGA transformations reduce image quality for all groups, with demographic-dependent drops; ANOVA rejects demographic independence for Realism datasets (p<0.05) but not SDv2.1 Celebrities.
- CLIP-I, DINO-I, and CLIP-directional correlate only weakly with human ratings (Spearman coefficients roughly 0.09 to 0.35).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SEGA semantic transformations | Realism and SDv2.1 generated faces · Realism Non-Celebrities, Realism Celebrities, and SDv2.1 Celebrities | image quality | Hurt |
| use different demographic groups | Realism face generation · Realism Non-Celebrities and Realism Celebrities | source image quality | Mixed |
| use different demographic groups | SDv2.1 face generation · SDv2.1 Celebrities | image quality | No clear effect |
| match training-data demographic proportions to the target distribution | generative models trained on biased data · Gaussian mixture analytical model | total variation distance to target distribution | Helped |
| use CLIP-I, DINO-I, or CLIP-directional as proxies for human ratings | generated face quality assessment · all three datasets | Spearman correlation with human assessment: 0.147; 0.107; 0.128; 0.197; 0.122; 0.348; 0.245; 0.142; 0.0908 | Hurt |
Surprising findings
- Demographic groups with higher source image quality were not those with higher transformation correctness, suggesting SEGA introduces its own bias.
- Embedding-based quality metrics show statistically significant but low correlation with human judgments, so they are poor proxies for perceived face quality.
Limitations
- Non-celebrity variations required manual filtering, and SDv2.1 non-celebrity data was dropped.
- SEGA transformations often fail to apply the requested attribute or degrade face quality.
- The analytical GMM model is a simplified proxy and its empirical validation uses a single StyleGAN on CelebA.
Experimental setup
Datasets: LFW CelebA LAION-5B
Benchmarks: LFW
Metrics: verification accuracy CLIP-I DINO-I CLIP-directional user-rated image quality transformation correctness Spearman correlation one-way ANOVA
Baselines: LFW natural faces Fair Diffusion InstructPix2Pix
Builds on: SEGA Stable Diffusion CLIP-directional Fair Diffusion
Closest papers by meaning
- MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts (NeurIPS 2024 · similarity 0.87)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.87)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.87)
- Fair Text-to-Image Diffusion via Fair Mapping (AAAI 2025 · similarity 0.87)
- High-fidelity Person-centric Subject-to-Image Synthesis (CVPR 2024 · similarity 0.87)
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.