Research map / Image generation / Text-to-image diffusion
Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns interpretable latent concept vectors in the Stable Diffusion U-Net h-space via self-supervised reconstruction, then adds them at inference to improve fairness, safety, and responsible text-guided generation.
Problem
Diffusion text-to-image models generate biased or harmful content, and existing h-space direction-discovery methods cannot find directions for arbitrary (especially ethical or safety) concepts without human interpretation, labeled data, or external attribute classifiers.
Approach
Optimize a concept vector added to the frozen diffusion U-Net bottleneck so that reconstructing images generated with a concept-specific prompt from a prompt missing that concept has minimal loss; the vector thus captures the missing concept without labels or external classifiers. Sampling or adding these vectors at inference then mitigates bias and unsafe content.
Key results
- On Winobias, the gender deviation ratio for the Winobias set drops from 0.68 (SD) and 0.22 (UCE) to 0.17 with the proposed method.
- On I2P, combining the anti-sexual and anti-violence vectors with SLD lowers the average inappropriate rate from 0.20 to 0.16; with SD it lowers it from 0.41 to 0.27.
- Image quality stays near the original SD (FID 15.98 vs 14.09; CLIP 31.03 vs 31.33 on COCO-30K).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use uniformly sampled learned gender vectors during generation | Stable Diffusion v1.4 · Winobias (Gender) | deviation ratio: 0.68 -> 0.17 | Helped |
| Use uniformly sampled learned gender vectors during generation | Stable Diffusion v1.4 · Winobias extended (Gender+) | deviation ratio: 0.70 -> 0.23 | Helped |
| Use uniformly sampled learned race vectors during generation | Stable Diffusion v1.4 · Winobias (Race) | deviation ratio: 0.56 -> 0.23 | Helped |
| Use uniformly sampled learned race vectors during generation | Stable Diffusion v1.4 · Winobias extended (Race+) | deviation ratio: 0.48 -> 0.20 | Helped |
| Add anti-sexual and anti-violence concept vectors | Stable Diffusion · I2P | proportion classified as inappropriate: 0.41 ±0.007 -> 0.27 ±0.006 | Helped |
| Add anti-sexual and anti-violence concept vectors | SLD · I2P | proportion classified as inappropriate: 0.20 ±0.006 -> 0.16 ±0.005 | Mixed |
| Add anti-sexual and anti-violence concept vectors | ESD · I2P | proportion classified as inappropriate: 0.32 ±0.007 -> 0.27 ±0.006 | Helped |
| Activate the ethical concept vector specified by the prompt | Stable Diffusion v1.4 · Responsible-concept prompts | deviation ratio (Gender): 0.2199 -> 0.1433 | Helped |
| Activate the ethical concept vector specified by the prompt | Stable Diffusion v1.4 · Responsible-concept prompts | deviation ratio (Race): 0.1600 -> 0.1399 | Helped |
| Activate the ethical concept vector specified by the prompt | Stable Diffusion v1.4 · Responsible-concept prompts | sexual content detection rate: 0.4300 -> 0.2640 | Helped |
| Activate the ethical concept vector specified by the prompt | Stable Diffusion v1.4 · Responsible-concept prompts | violent content detection rate: 0.4551 -> 0.3204 | Helped |
| Add safety concept vectors | Stable Diffusion · COCO-30K | FID: 14.09 -> 15.98 | Hurt |
Limitations
- Optimizing a single safety vector for a broad concept such as 'safety' is difficult, so per-concept vectors are used instead.
- Abstract concepts such as 'hate' improve safety but degrade image quality.
- Evaluation relies on CLIP, NudeNet and Q16 classifiers, which have their own reliability limits.
Experimental setup
Datasets: Winobias I2P COCO-30K
Benchmarks: Winobias I2P COCO-30K
Metrics: deviation ratio attack/inappropriate proportion via NudeNet and Q16 FID CLIP score
Baselines: SD UCE SLD ESD
Builds on: h-space semantic directions (Kwon et al.) unified concept editing (UCE) safe latent diffusion (SLD) erased stable diffusion (ESD) classifier-free guidance
Closest papers by meaning
- Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimization (CVPR 2025 · similarity 0.89)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.88)
- Unveiling Concept Attribution in Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Trade-Offs in Fine-Tuned Diffusion Models between Accuracy and Interpretability (AAAI 2024 · similarity 0.87)
- Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control (CVPR 2025 · similarity 0.87)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.