Research map / Image generation / Text-to-image diffusion

RespoDiff: Dual-Module Bottleneck Transformation for Responsible & Faithful T2I Generation

NeurIPS 2025 · Image generation › Text-to-image diffusion · approach: T2I evaluation and controllable adapters · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RespoDiff, which adds two learnable bottleneck (h-space) transformations to a frozen Stable Diffusion UNet, one for responsible concept alignment and one for semantic alignment, coordinated by score-matching losses, to improve fairness and safety without hurting fidelity.

Problem

Text-to-image diffusion models reproduce gender/racial stereotypes and produce unsafe content, and existing mitigation methods often degrade semantic fidelity and image quality.

Approach

Learn two independent bottleneck transformations: a Responsible Concept Alignment Module that shifts the neutral-prompt UNet trajectory toward a target concept via a score-matching loss against the target-prompt prediction, and a Semantic Alignment Module that regularizes the full transformation to match the original neutral-prompt prediction, so debiasing does not drift from the prompt.

Key results

• On Winobias gender with SD v1.4, RespoDiff reaches DevRat 0.14 vs 0.68 for SD and 0.17 for SDisc, with WinoAlign 27.30 and FID 14.91.
• On Winobias race, RespoDiff achieves DevRat 0.16 vs 0.56 for SD, with the best FID (12.82) among compared methods.
• On I2P safety, RespoDiff reaches 0.16 vs 0.27 for SD (lower is better), while SLD gets 0.20 and ESD 0.32.

Measured effects reported in the paper

Surprising findings

• The shared single-transformation variant gives a lower deviation ratio than the separate-module design, which the authors attribute to dedicated transformations, not to the separate modules as expected.
• Increasing lambda to 4 yields the best FID and CLIP but a much worse deviation ratio (0.29), showing a fairness-fidelity trade-off.

Limitations

• Relies on a predefined set of binary gender and limited racial concepts, so non-binary or intersectional identities are not handled.
• Assumes a well-specified neutral prompt ('a person', 'a scene') chosen in advance.
• Residual biases may remain where fairness and neutrality are not clear-cut; results averaged over only 2 seeds.

Experimental setup

Datasets: Winobias Gender+ Winobias Race+ Winobias I2P COCO-30K

Benchmarks: Winobias Gender+ Winobias Race+ Winobias I2P COCO-30K

Metrics: DevRat WinoAlign FID CLIP score Q16/NudeNet accuracy

Baselines: SD SDisc FDF BAct FD SLD ESD

Builds on: SDisc h-space concept discovery classifier-free guidance Latent diffusion models

Closest papers by meaning

• Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
• Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control (CVPR 2025 · similarity 0.88)
• Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.88)
• OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.87)
• T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.87)
• Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.87)