Research map / Image generation / Text-to-image diffusion
PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoGDiff, a fine-tuning objective for diffusion models on imbalanced text-image data that replaces the ground-truth target with a product of Gaussians combining the target and the prediction conditioned on a neighboring text embedding.
Problem
Diffusion models fine-tuned on imbalanced image-text datasets generate accurate, diverse images for majority groups but perform poorly for minority groups with few samples.
Approach
Replace the ground-truth denoising target with a product of Gaussians built from the true target and the model's prediction under a neighboring text embedding, weighted by image similarity and inverse text density (estimated via a VAE ELBO), so minority samples borrow statistical strength from neighbors.
Key results
- On AgeDB-IT2I-L, PoGDiff reduces all-shot FID from 7.67 (Vanilla) to 6.03 and raises few-shot DINO from 0.25 to 0.52.
- Human identity score on AgeDB-IT2I-S few-shot rises from 0.00 (Vanilla) to 1.00 for PoGDiff.
- gRecall on AgeDB-IT2I-S few-shot reaches 1.000 for PoGDiff versus 0.000 for Vanilla and CBDM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add neighboring embedding y' and PoG training objective (ablation) | Stable Diffusion v1.5 fine-tuning · AgeDB-IT2I-Large, few-shot | FID: 11.67 -> 10.16 | Helped |
| add image similarity weighting ψ_img-sim (ablation) | PoGDiff · AgeDB-IT2I-Large, few-shot | FID: 10.49 -> 10.16 | Helped |
| add inverse text density weighting ψ_inv-txt-den (ablation) | PoGDiff · AgeDB-IT2I-Large, few-shot | FID: 10.43 -> 10.16 | Helped |
| fine-tune with PoGDiff instead of Vanilla | Stable Diffusion v1.5 · AgeDB-IT2I-Large, few-shot | DINO: 0.25 -> 0.52 | Helped |
| fine-tune with PoGDiff instead of Vanilla | Stable Diffusion v1.5 · AgeDB-IT2I-Large, few-shot | Human Score: 0.20 -> 0.68 | Helped |
| fine-tune with PoGDiff instead of Vanilla | Stable Diffusion v1.5 · AgeDB-IT2I-Large, few-shot | gRecall: 0.200 -> 0.540 | Helped |
Surprising findings
- CBDM performs extremely poorly on small AgeDB subsets, attributed to sampling text conditions from the whole space, which misguides training with natural-language prompts.
- Vanilla Stable Diffusion and T2H achieve near-zero few-shot human identity scores on AgeDB-IT2I-S, showing standard fine-tuning fails completely for minority identities.
Limitations
- Relies on borrowing statistical strength from minority neighbors, requiring at least 2 images per minority class; sensitive to minority set size.
- Image similarity and inverse density weights depend on a pretrained image encoder and a trained VAE; producing them for arbitrary datasets is open.
- Evaluation limited to the target fine-tuning dataset; color deviation persists and is not addressed by the method.
Experimental setup
Datasets: AgeDB DigiFace VGGFace2 CIFAR-100-LT
Benchmarks: AgeDB-IT2I DigiFace-IT2I VGGFace-IT2I CIFAR-100-IT2I
Metrics: FID DINO score human identity score GPT-4o similarity score gRecall
Baselines: Vanilla fine-tuning CBDM T2H
Builds on: Stable Diffusion Class Balancing Diffusion Model (CBDM) Long-Tailed Diffusion Model with Oriented Calibration (T2H) Deep Imbalanced Regression (DIR) VIR
Closest papers by meaning
- DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.87)
- LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders (NeurIPS 2025 · similarity 0.86)
- A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization (NeurIPS 2024 · similarity 0.86)
- Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.86)
- Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion (CVPR 2025 · similarity 0.86)
- RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.