Research map / Image generation / Text-to-image diffusion

Efficient bias mitigation in T2I diffusion models using Concept Graphs

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Inference-time T2I improvement · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CO-ALIGN, which extracts concept graphs from the text encoder and denoiser of Stable Diffusion v1.5 and jointly aligns them toward a target topology to reduce occupational demographic bias, and also uses the resulting neighbourhood propagation to improve robustness of post-hoc nudity unlearning.

Problem

Text-to-image diffusion models reproduce demographic bias from training data, and existing debiasing methods that edit only the text encoder or only the denoiser either leave bias intact or produce semantically incoherent images.

Approach

Treat bias as asymmetry in the relational structure of concept representations and align a concept graph (built from text-encoder embedding cosine similarity and denoiser cross-attention concept-neuron fingerprints) toward a target topology with a differentiable soft-rank loss, applied sequentially to the text encoder and then the denoiser.

Key results

• Achieves FD 0.032 for gender (30.4% better than next best) and FD 0.043 for race (75.4% better), with FID 98.88 for gender.
• Reduces semantically incoherent outputs by 88% relative to the base model per LLaVA classification.
• With ESD unlearning, average nudity rate drops from 10.0% to 0.8% on adversarial prompts after CO-ALIGN pre-alignment.

Surprising findings

• Text-encoder-only alignment alone produces semantically incoherent outputs, which denoiser alignment subsequently resolves.
• Unsupervised neighbouring concepts shift toward the unlearning target after aligning only the adversarial concepts (neighbourhood pulling effect).

Limitations

• Denoiser graph extraction requires a full denoising trajectory per concept per step, making UNet alignment compute-intensive.
• Evaluated only on Stable Diffusion v1.5; not validated on DiT or SDXL architectures.
• Incoherence and nudity judged by a single VLM (LLaVA) rather than human raters.

Experimental setup

Datasets: MS-COCO LAION-5B (analysis only) I2P

Benchmarks: FairFace-based fairness evaluation on occupation prompts (nurse, doctor, fire fighter, receptionist) I2P adversarial nudity concepts

Metrics: Fairness Discrepancy FID CLIP-I CLIP-T incoherence rate nudity flag rate

Baselines: Asyrp H-Distribution Latent Direction Finetuning DiffLens UCE Debias-VL ESD SAFREE

Builds on: TRUST DiffLens UCE ESD SAFREE

Closest papers by meaning

• CASL: Concept-Aligned Sparse Latents for Interpreting Diffusion Models (NeurIPS 2026 · similarity 0.88)
• You Can’t Have It Both Ways: Concept Entanglement Limits Diffusion Model Unlearning (NeurIPS 2026 · similarity 0.87)
• Towards Transformer-Based Aligned Generation with Self-Coherence Guidance (CVPR 2025 · similarity 0.87)
• Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.87)
• CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching (NeurIPS 2024 · similarity 0.86)
• Information Theoretic Text-to-Image Alignment (ICLR 2025 · similarity 0.86)