Research map / Image generation / Text-to-image diffusion
LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LightFair, which debiases only the CLIP text encoder of Stable Diffusion via LoRA fine-tuning with distance-constrained losses and applies the debiased encoder only in later sampling steps, reaching SOTA gender/race debiasing at lower training cost.
Problem
Text-to-image diffusion models produce demographically biased images (gender, race, age); existing debiasing methods either fine-tune the full diffusion model with heavy compute or add auxiliary networks/reference data that slow sampling.
Approach
Treat the pre-trained text encoder as the lightweight, front-end locus of bias: enforce equalized odds and equalized quality via distance constraints between text embeddings and CLIP image-embedding attribute centroids (with cross-attention foreground extraction), and switch to the debiased encoder only at later denoising steps, where attribute information emerges.
Key results
- On SD v1.5 gender debiasing, LightFair reduces Bias-O from 0.73 to 0.30 and Bias-Q from 1.90 to 0.99 while improving CLIP-T to 30.57 and FID to 233.53 (vs. 275.13 for SD).
- On SD v1.5 race debiasing, Bias-O drops from 0.54 to 0.18 and FID improves to 259.96 vs. 275.13 for the original SD.
- Abstract claims SOTA debiasing on SD v1.5 at about 1/4 of the training burden with virtually no added sampling cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the text encoder with LightFair | Stable Diffusion v1.5 · Gender | Bias-O: 0.73 (±0.05) -> 0.30 (±0.08) | Helped |
| fine-tune the text encoder with LightFair | Stable Diffusion v1.5 · Race | Bias-O: 0.54 (±0.02) -> 0.18 (±0.04) | Helped |
| fine-tune the text encoder with LightFair | Stable Diffusion v2.1 · Gender | Bias-O: 0.85 (±0.05) -> 0.33 (±0.10) | Helped |
| fine-tune the text encoder with LightFair | Stable Diffusion v2.1 · Race | Bias-O: 0.63 (±0.01) -> 0.40 (±0.03) | Helped |
| fine-tune the text encoder with LightFair | Stable Diffusion v1.5 · Gender | CLIP-T: 29.31 (±0.06) -> 30.57 (±0.16) | Helped |
| fine-tune the text encoder with LightFair | Stable Diffusion v2.1 · Gender | CLIP-T: 29.90 (±0.15) -> 30.82 (±0.19) | Helped |
| add the odds loss ℓo (ablation) | LightFair · Gender, Stable Diffusion v1.5 | Bias-O: 0.70 (±0.03) -> 0.67 (±0.08) | Helped |
| add the quality loss ℓq to ℓo (ablation) | LightFair · Gender, Stable Diffusion v1.5 | Bias-Q: 1.07 (±0.46) -> 0.93 (±0.46) | Helped |
| add the regularization loss ℓreg to ℓo (ablation) | LightFair · Gender, Stable Diffusion v1.5 | CLIP-T: 30.45 (±0.11) -> 28.38 (±0.11) | Hurt |
| add the regularization loss ℓreg to ℓo and ℓq (ablation) | LightFair · Gender, Stable Diffusion v1.5 | Bias-O: 0.56 (±0.08) -> 0.45 (±0.09) | Helped |
| add adaptive foreground extraction (ablation) | LightFair with ℓo, ℓq, and ℓreg · Gender, Stable Diffusion v1.5 | CLIP-T: 30.29 (±0.44) -> 32.19 (±0.07) | Helped |
| add adaptive foreground extraction (ablation) | LightFair with ℓo, ℓq, and ℓreg · Gender, Stable Diffusion v1.5 | Bias-O: 0.45 (±0.09) -> 0.34 (±0.10) | Helped |
Surprising findings
- The text encoder's bias is amplified by the noise prediction network during denoising, so debiasing the text encoder alone propagates to images.
- Attribute information emerges only in later denoising stages, so intervening with the debiased encoder early harms quality while late intervention preserves it.
Limitations
- Evaluated only on gender and race (plus age in extension) with occupation prompts; attribute classifiers used for bias measurement may carry their own biases.
- Debiasing reduces CLIP-I and some quality metrics relative to some baselines (e.g., UCE on CLIP-I), a stated trade-off.
Experimental setup
Datasets: LAION-Aesthetics V2
Benchmarks: gender and race bias evaluation on occupation prompts (Bias-O, Bias-Q)
Metrics: Bias-O Bias-Q CLIP-T CLIP-I FID IS AS-R AS-A
Baselines: FairD UCE FinetuneFD FairMapping BalancingAct debias VL
Builds on: Stable Diffusion CLIP LoRA Fair Diffusion
Closest papers by meaning
- A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization (NeurIPS 2024 · similarity 0.91)
- Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion (CVPR 2025 · similarity 0.89)
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.89)
- Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Fair Text-to-Image Diffusion via Fair Mapping (AAAI 2025 · similarity 0.88)
- One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.