Research map / Image generation / Text-to-image diffusion
Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a plug-and-play method that distills a Stable Diffusion text encoder (RICE) and U-Net latent space (RIIDL) into concept-conditioned add-on modules with concept whitening, enabling multi-facet fairness and safety control for T2I generation.
Problem
Responsible text-to-image generation requires controlling fairness (age, gender, race) and safety concepts jointly, but existing methods handle facets individually, lack interpretability, and often modify the base model, degrading performance.
Approach
Distill the frozen T2I pipeline's text encoder and diffusion U-Net into external add-on modules conditioned on a set of responsible concepts, apply concept whitening to decorrelate the learned concept subspaces, and modulate embeddings and latents additively at inference across both plug-in points.
Key results
- On WinoBias gender deviation ratio, the method reaches values such as 0.04 for CEO and Nurse versus 0.92 and 1.00 for SD v1.4.
- On I2P, average inappropriate proportion drops from 0.41 (SD) to 0.20, matching SLD (0.20) and beating ESD (0.32) and Vector-SD (0.27).
- Averaged over 35 WinoBias professions, deviation ratio is 0.05 versus 0.31 for the best prior baseline (TIMEP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method instead of Original-SD | Stable Diffusion · WinoBias (35-profession average) | deviation ratio: 0.67 ± 0.01 -> 0.05 ± 0.02 | Helped |
| Use the proposed method instead of TIMEP | Stable Diffusion · WinoBias (35-profession average) | deviation ratio: 0.31 ± 0.00 -> 0.05 ± 0.02 | Helped |
| Use the proposed method instead of SD | Stable Diffusion · I2P (average) | inappropriate-image proportion: 0.41 -> 0.20 | Helped |
| Use the proposed method instead of SLD | Stable Diffusion · I2P (average) | inappropriate-image proportion: 0.20 vs 0.20 | No clear effect |
Limitations
- Requires tuning multiple hyperparameters (lambda weights) for control.
- Could be misused by substituting the responsible concept set with irresponsible concepts.
Experimental setup
Datasets: WinoBias I2P
Benchmarks: WinoBias Gender-Pro Race-Pro I2P
Metrics: deviation ratio proportion of inappropriate images
Baselines: SD SLD ESD UCE Vector-SD Concept Algebra Debias-VL TIME TIMEP
Builds on: concept whitening knowledge distillation Stable Diffusion Vector-SD
Closest papers by meaning
- RespoDiff: Dual-Module Bottleneck Transformation for Responsible & Faithful T2I Generation (NeurIPS 2025 · similarity 0.88)
- Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.87)
- Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimization (CVPR 2025 · similarity 0.87)
- T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation (CVPR 2025 · similarity 0.86)
- SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.