Research map / Image generation / Fast diffusion sampling
Balancing Act: Distribution-Guided Debiasing in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Debiases pretrained diffusion models at inference time by jointly guiding a batch's h-space features so the generated attribute distribution matches a user-specified reference distribution, using a linear attribute distribution predictor.
Problem
Diffusion models reflect dataset biases (e.g., gender, race, eyeglasses imbalance in faces), and existing debiasing methods require reference datasets or model retraining; the paper targets fair generation without retraining, given only a reference attribute distribution.
Approach
Instead of forcing each sample to a preset attribute (sample guidance), jointly denoise a batch and guide the h-space bottleneck features with a linear Attribute Distribution Predictor so the batch's aggregated attribute softmax matches the reference distribution, with the ADP trained on pseudo-labels from existing attribute classifiers.
Key results
- On CelebA-HQ unconditional P2 model, H-Distribution Guidance reduces gender FD from 0.178 (random sampling) to 0.049 with FID 50.27, and eyeglasses FD from 0.251 to 0.014.
- On Stable Diffusion v1.5 for doctor prompts, gender FD drops from 0.317 to 0.024 (FID 70.69), outperforming ITI-Gen (0.049) and Fair Diffusion (0.227).
- Balanced training with distribution-guided generated data raises Black-female accuracy from 75.76 to 91.04 and Black-male accuracy from 70.29 to 90.72 for a race classifier.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use distribution guidance instead of random sampling | P2 · Gender | FD: 0.178 -> 0.049 | Helped |
| Use distribution guidance instead of random sampling | P2 · Gender | FID: 54.59 -> 50.27 | Helped |
| Use distribution guidance instead of random sampling | P2 · Race | FD: 0.334 -> 0.113 | Helped |
| Use joint distribution guidance for both attributes | P2 · Gender + Eyeglasses | FD: 0.214 -> 0.057 | Helped |
| Use distribution guidance toward a skewed reference distribution | P2 · 0.20F - 0.80M | FID: 72.26 -> 51.65 | Helped |
| Use distribution guidance on Stable Diffusion v1.5 | Stable Diffusion v1.5 · Doctor | FD: 0.355 -> 0.015 | Helped |
| Use distribution guidance on Stable Diffusion v1.5 | Stable Diffusion v1.5 · Firefighter | FD: 0.235 -> 0.093 | Helped |
| Use batch size 100 rather than batch size 2 (ablation) | H-Distribution Guidance · Gender balancing | FD: 0.108 -> 0.046 | Helped |
| Use batch size 100 rather than batch size 2 (ablation) | H-Distribution Guidance · Gender balancing | FID: 60.12 -> 51.64 | Mixed |
Surprising findings
- Latent editing in h-space achieves the best FD for gender (0.001) but produces visibly collapsed images, so FD alone is misleading.
- Sample-level classifier guidance degrades quality because forcing per-sample attribute changes distorts late-stage features, whereas batch-level distribution guidance transforms mainly samples near the decision boundary.
Limitations
- Requires training h-space attribute classifiers and labels from existing attribute classifiers, so accuracy depends on those classifiers.
- Still needs per-model h-space classifier training; not applicable without attribute classifiers or labeled data.
- Evaluated mainly on face attributes and a few text prompts (doctor, firefighter).
Experimental setup
Datasets: CelebA-HQ FFHQ WaterBirds LAION
Benchmarks: CelebA-HQ FFHQ
Metrics: Fairness Discrepancy (FD) FID
Baselines: Random Sampling Universal Guidance Latent Editing MaGNET ITI-Gen Fair Diffusion H-Sample Guidance
Builds on: classifier guidance h-space (Kwon et al. diffusion semantics) DDIM Universal Guidance
Closest papers by meaning
- Diffusing DeBias: Synthetic Bias Amplification for Model Debiasing (NeurIPS 2025 · similarity 0.88)
- Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models (ICLR 2025 · similarity 0.88)
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.88)
- Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion (CVPR 2025 · similarity 0.87)
- Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales (NeurIPS 2026 · similarity 0.87)
- Don't Play Favorites: Minority Guidance for Diffusion Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.