Research map / Image generation / Text-to-image diffusion
Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RoleBench to measure directional role bias in action-based T2I generation (e.g., "mouse chasing cat") and ReBind, which LoRA-fine-tunes SDXL on LLM-selected active/passive intermediate compositions to reduce role collapse.
Problem
Text-to-image diffusion models default to frequent reversed role relations (e.g., "cat chasing mouse") when asked for rare action compositions such as "mouse chasing cat", a failure called role collapse.
Approach
Rare compositions are decomposed into active intermediates (rare subject kept in its agent role with a new patient) and passive intermediates (rare patient kept in its patient role with a new agent); fine-tuning on these intermediate images with LoRA reinforces the underrepresented role bindings without inference-time overhead or architectural change.
Key results
- Pre-trained T2I models show positive directional bias beta on rare compositions across all models; SDXL bias is 23.1 on VQAScore for rare prompts.
- ReBind (SDXL, lambda=5) reduces VQAScore bias from 23.1 to 7.9 on RoleBench.
- Human raters preferred ReBind over RPG (73%), R2F (74%), RRNet (78%), DALL-E 3 (73%), and SDXL (69%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SDXL on active and passive intermediate compositions | SDXL · RoleBench rare compositions | VQAScore β: 23.1 -> 7.9 | Helped |
| Add active intermediates to passive-intermediate training (ablation) | ReBind · RoleBench rare compositions | VQAScore β: 10.1 -> 7.9 | Helped |
| Add passive intermediates to active-intermediate training (ablation) | ReBind · RoleBench rare compositions | VQAScore β: 9.6 -> 7.9 | Helped |
| Increase active-intermediate weight λ from 1 to 5 (ablation) | ReBind · RoleBench rare compositions | VQAScore β: 12.0 -> 7.9 | Helped |
| Fine-tune directly on frequent and rare images instead of intermediate compositions (ablation) | SDXL · RoleBench rare compositions | VQAScore β: 7.9 -> 14.7 | Hurt |
| Fine-tune on intermediate compositions | SDXL · COCO 2017 validation | FID: 83.95 -> 90.56 | Hurt |
| Fine-tune on intermediate compositions | SDXL · COCO 2017 validation | CLIPScore: 21.17 -> 21.20 | No clear effect |
Surprising findings
- Fine-tuning directly on DALL-E 3 rare/frequent pairs (FT Freq/Rare) still collapses to the frequent composition, since both prompts yield frequent-direction images.
- Most compositional generation methods (IterComp, RPG, R2F, RRNet) yield little or no bias reduction relative to SDXL, and SLD shows score hacking via GPT-4o feedback editing.
Limitations
- Generalization to unseen, semantically distant (OOD) relations remains limited (bias 36.4 vs 38.3 for SDXL).
- Benchmark is small: 10 relations restricted to animated objects, with one rare/frequent pair per relation.
- Fine-tuning causes some drop in global VQAScore on frequent prompts and a higher FID on COCO (83.95 to 90.56).
Experimental setup
Datasets: RoleBench COCO 2017 validation
Benchmarks: RoleBench COCO 2017 validation (5k images)
Metrics: VQAScore role direction bias beta spatial/orientation/pose/facial expression VQA scores CLIPScore FID human preference
Baselines: SDXL SD3 SD3.5 AuraFlow2 DALL-E 3 SLD RPG IterComp R2F RRNet FT Freq/Rare
Builds on: VQAScore Stable Diffusion XL RRNet RPG SLD
Closest papers by meaning
- Improving Compositional Generation with Diffusion Models Using Lift Scores (ICML 2025 · similarity 0.89)
- IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation (ICLR 2025 · similarity 0.89)
- How Diffusion Models Learn to Factorize and Compose (NeurIPS 2024 · similarity 0.87)
- Diffusion Classifiers Understand Compositionality, but Conditions Apply (NeurIPS 2025 · similarity 0.87)
- How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.87)
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.