Research map / Image generation / GANs
Diverse Rare Sample Generation with Pretrained GANs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates diverse rare yet realistic images from pretrained StyleGAN2/StyleGAN2-ADA by multi-start gradient optimization of latent vectors, with rarity (normalizing-flow likelihood), similarity, and diversity objectives.
Problem
Pretrained GANs rarely produce rare samples in low-density regions, and existing methods either lose diversity or ignore rare samples; generating diverse rare variants of a reference image without retraining the GAN is the target.
Approach
Optimize latent vectors of a frozen GAN with a multi-objective loss: minimize normalizing-flow log-likelihood in VGG16-fc2 feature space (rarity), penalize leaving a k'-NN similarity ball around the reference, and push multiple multi-start samples apart (diversity), constrained to the real k-NN manifold.
Key results
- On FFHQ-StyleGAN2, the method raises Rarity Score from 18.88 (baseline) to 23.50 and LPIPS from 0.73 to 0.76, with precision rising to 0.92 versus 0.69 for baseline, though FID worsens to 7.38 from 4.17.
- On AFHQ Dog with StyleGAN2-ADA, Rarity Score improves from 21.19 to 27.32 and precision from 0.77 to 0.85.
- Normalizing-flow log-likelihood correlates with k-NN-based density at Pearson 0.928 for real samples and 0.815 for fake samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-objective rare-sample optimization instead of baseline sampling | StyleGAN2 · FFHQ | RS: 18.88 -> 23.50 | Helped |
| Use multi-objective rare-sample optimization instead of baseline sampling | StyleGAN2 · FFHQ | Precision: 0.69 -> 0.92 | Helped |
| Use multi-objective rare-sample optimization instead of baseline sampling | StyleGAN2 · FFHQ | FID: 4.17 -> 7.38 | Hurt |
| Use multi-objective rare-sample optimization instead of baseline sampling | StyleGAN2-ADA · AFHQ Cat | RS: 17.47 -> 20.71 | Helped |
| Use multi-objective rare-sample optimization instead of baseline sampling | StyleGAN2-ADA · AFHQ Dog | RS: 21.19 -> 27.32 | Helped |
| Use multi-objective rare-sample optimization instead of baseline sampling | StyleGAN2-ADA · MetFaces | RS: 16.90 -> 21.25 | Helped |
| Add similarity regularization to the rarity-only objective (ablation) | FFHQ-StyleGAN2 objective · FFHQ | RS: 18.99 -> 21.11 | Helped |
| Add similarity regularization to the rarity-only objective (ablation) | FFHQ-StyleGAN2 objective · FFHQ | LPIPS: 0.752 -> 0.766 | Helped |
| Add the diversity term to the rarity + similarity objective (ablation) | FFHQ-StyleGAN2 objective · FFHQ | RS: 21.11 -> 21.28 | Helped |
| Add the diversity term to the rarity + similarity objective (ablation) | FFHQ-StyleGAN2 objective · FFHQ | LPIPS: 0.766 -> 0.768 | Helped |
| Replace L2 distance with L1 distance (ablation) | FFHQ-StyleGAN2 method · FFHQ | RS: 24.02 -> 24.12 | No clear effect |
| Replace L2 distance with cosine similarity (ablation) | FFHQ-StyleGAN2 method · FFHQ | RS: 24.02 -> 24.00 | No clear effect |
Surprising findings
- Polarity sampling achieves higher rarity score and recall than the proposed method on FFHQ but has very low precision (0.38-0.39) and much worse FID (about 33-34), as its samples often fall outside the real manifold.
- Some attributes that are already common in the dataset (e.g., BigNose, NoBeard) increase with the method, attributed to dependence on the NF likelihood.
Limitations
- Results depend on the GAN's capabilities and require an additional density estimator.
- Some optimized samples show artifacts and low fidelity, since the k-NN manifold overestimates the real support.
- Alters multiple attributes simultaneously, limiting controlled manipulation.
Experimental setup
Datasets: FFHQ AFHQ Cat AFHQ Dog MetFaces ImageNet
Benchmarks: FFHQ AFHQ Cat AFHQ Dog MetFaces
Metrics: Rarity Score precision recall LPIPS FID KID attribute percentage
Baselines: Polarity sampling StyleGAN2 baseline sampling StyleGAN2-ADA baseline sampling
Builds on: Glow Polarity sampling rarity score StyleGAN2-ADA Model Soups
Closest papers by meaning
- RareGAN: Generating Samples for Rare Classes (AAAI 2022 · similarity 0.90)
- Normalized Diversification (CVPR 2019 · similarity 0.90)
- Diverse Image Generation via Self-Conditioned GANs (CVPR 2020 · similarity 0.89)
- Consistency-GAN: Training GANs with Consistency Model (AAAI 2024 · similarity 0.87)
- RG-GAN: Dynamic Regenerative Pruning for Data-Efficient Generative Adversarial Networks (AAAI 2024 · similarity 0.87)
- MS$^3$D: A RG Flow-Based Regularization for GAN Training with Limited Data (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.