Research map / Image generation / Autoregressive and latent generation
Quality-Diversity Generative Sampling for Learning with Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QDGS, a model-agnostic quality-diversity latent-space sampler that uses CLIP language prompts as objective and measures to produce balanced synthetic datasets from biased generators, showing debiasing on colored shapes and improved fairness in facial recognition.
Problem
Synthetic training data from generative models inherit representational biases and cover only dominant modes; existing debiasing sampling methods remove one bias at a time and cannot produce intersectional balance.
Approach
Apply quality-diversity optimization (CMA-MAEGA latent space illumination) in a generator's latent space, with objective and diversity measures defined by CLIP similarity to user-written language prompts, so that multiple attributes are balanced jointly without fine-tuning the generator.
Key results
- On CASIA-trained models, QDGS pretraining raised dark-skinned RFW accuracy from 88.08% (no pretraining) to 88.94% (QD50).
- QDGS increased the proportion of images recognized with dark skin tones from 9.4% to 25.2% and old-age images from 37.0% to 45.7% versus random sampling.
- QDGS debiasing of color-biased shape classifiers improved accuracy by up to about 27%, with larger gains at stronger biases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace random 50K synthetic pretraining with QDGS 50K pretraining | ResNet101 trained on CASIA · RFW dark-skinned | accuracy: 88.69\pm0.10 -> 88.94\pm0.07 | Helped |
| Replace random 50K synthetic pretraining with QDGS 50K pretraining | ResNet101 trained on CASIA · Facial recognition benchmarks | average accuracy: 94.89\pm0.03 -> 94.92\pm0.02 | Helped |
| Use QDGS pretraining instead of no synthetic pretraining | ResNet101 trained on CASIA · Facial recognition benchmarks | average accuracy: 94.52\pm0.03 -> 94.92\pm0.02 | Helped |
| Replace random synthetic sampling with QDGS sampling | StyleGAN2 synthetic facial dataset · Synthetic image distribution | dark-skin image proportion: 9.4% -> 25.2% | Helped |
| Replace random synthetic sampling with QDGS sampling | StyleGAN2 synthetic facial dataset · Synthetic image distribution | old-image proportion: 37.0% -> 45.7% | Helped |
| Train with QDGS-generated balanced data instead of the original biased data | Color-biased shape classifier · Balanced shape evaluation set | accuracy: up to $\approx27\%$ | Helped |
| Use QDGS pretraining with ArcFace rather than ArcFace baseline alone (ablation) | Facial recognition model · RFW dark-skinned | accuracy: 79.7\pm0.3 -> 82.8\pm0.2 | Helped |
| Add QDGS pretraining to AdaFace (ablation) | Facial recognition model · RFW dark-skinned | accuracy: 88.1\pm0.1 -> 88.9\pm0.1 | Helped |
| Use QDGS 50K pretraining instead of random 50K pretraining | ResNet101 trained on BUPT · Facial recognition benchmarks | average accuracy: 96.31\pm0.03 -> 96.30\pm0.02 | No clear effect |
| Use more detailed measure prompts (ablation) | QDGS shape sampling · Shape measure-space coverage | coverage smoothness | Helped |
| Use qualitative age prompts instead of quantitative age prompts (ablation) | QDGS facial sampling · Synthetic facial images | age appropriateness | Hurt |
Surprising findings
- Some seemingly benign quality prompts (e.g., 'well-lit', 'HD', 'photorealistic') introduced unintended visual biases such as lighter skin tones or bokeh backgrounds.
- Facial rotation remained highly skewed in QD50 despite the balancing, showing that unanticipated attributes are not controlled.
Limitations
- Balancing with uniform distributions may be inappropriate when proportional representation of the target population is needed.
- Unanticipated attributes cannot be enumerated a priori, and generator-inherent biases limit what concepts can be generated.
- Language prompts can introduce linguistic biases and unintended visual associations.
Experimental setup
Datasets: FFHQ CASIA-WebFace BUPT-Balancedface IJB-C CelebA RFW LFW CFP-FP CPLFW CALFW AgeDB color-biased shapes
Benchmarks: RFW LFW CFP-FP CPLFW CALFW AgeDB color-biased shapes balanced evaluation set
Metrics: accuracy disparate impact ratio attribute proportion distribution
Baselines: Random sampling (Rand15/Rand50) ArcFace AdaFace No pretraining
Builds on: CMA-MAEGA latent space illumination CLIP AdaFace
Closest papers by meaning
- On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.85)
- Debiasing Synthetic Data Generated by Deep Generative Models (NeurIPS 2024 · similarity 0.85)
- Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory (ICLR 2025 · similarity 0.85)
- Image Generation Diversity Issues and How to Tame Them (CVPR 2025 · similarity 0.85)
- Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment (NeurIPS 2024 · similarity 0.84)
- Position: All Current Generative Fidelity and Diversity Metrics are Flawed (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.