Research map / Image generation / Text-to-image diffusion
Relation Rectification in Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Relation Rectification, training a lightweight heterogeneous GCN (RRNet) to adjust CLIP [EOT] text embeddings of object-swapped prompts so Stable Diffusion 2-1 generates the correct directional relation, with the diffusion model and text encoder frozen.
Problem
Text-to-image diffusion models often fail to depict directional relationships between objects (e.g., "a book on a bowl" yields "a bowl on a book") because the text encoder's [EOT] embeddings for object-swapped prompts are nearly identical.
Approach
Model object-swapped prompts as directed heterogeneous graphs (object, relation, and [EOT] node types), use an HGCN to produce an adjustment vector added to the [EOT] embedding, and train it with a denoising loss plus a negative loss on the swapped prompt, keeping SD and the text encoder frozen.
Key results
- RRNet (λ=1) raises positional relation accuracy under Qwen-VL-Chat from 0.467 (Stable Diffusion) to 0.729, and under LLaVA from 0.542 to 0.724.
- FID worsens from 83.73 (Stable Diffusion) to 110.01 at λ=1, indicating a fidelity cost that λ=0.4–0.6 mitigates.
- In a 63-person user study, RRNet (λ=0.6) was preferred 75.56% of the time versus 16.98% for Stable Diffusion and 7.46% for personalized diffusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RRNet to adjust the [EOT] embedding instead of vanilla conditioning | Stable Diffusion 2-1 · Positional relationships | accuracy (Qwen-VL-Chat): 0.467 -> 0.697 | Helped |
| use RRNet to adjust the [EOT] embedding instead of vanilla conditioning | Stable Diffusion 2-1 · Action relationships | accuracy (LLaVA): 0.543 -> 0.632 | Helped |
| increase RRNet adjustment weight from λ=0.6 to λ=1 (ablation) | RRNet · Action relationships | accuracy (Qwen-VL-Chat): 0.500 -> 0.490 | No clear effect |
| increase RRNet adjustment weight from λ=0.6 to λ=1 (ablation) | RRNet · COCO validation set | FID: 100.78 -> 110.01 | Hurt |
| add HGCN relational graph modeling (ablation) | RRNet · Positional relationships | accuracy (Qwen-VL-Chat): 0.509 -> 0.697 | Helped |
| add negative loss (ablation) | RRNet · Action relationships | accuracy (Qwen-VL-Chat): 0.451 -> 0.500 | Helped |
| add object node disentanglement (ablation) | RRNet · Action relationships | accuracy (Qwen-VL-Chat): 0.426 -> 0.500 | Helped |
| optimize the CLIP text encoder rather than use HGCN-based RRNet | Stable Diffusion 2-1 · Action relationships | accuracy (Qwen-VL-Chat): 0.399 -> 0.273 | Hurt |
Surprising findings
- Masking out the [EOT] embedding destroys relation semantics in generated images, while masking word embeddings causes only marginal change.
- Personalized diffusion (tuning the CLIP text encoder) underperforms vanilla Stable Diffusion on action relations under Qwen-VL-Chat (0.273 vs 0.399).
Limitations
- Image fidelity decreases as the adjustment weight λ grows (FID rises to 110 at λ=1).
- Evaluation relies on VLM chatbots (Qwen-VL-Chat, LLaVA) as judges and a self-curated benchmark of 21 relationships.
Experimental setup
Datasets: Relation Rectification Benchmark (self-curated) COCO validation set (for FID)
Benchmarks: Relation Rectification Benchmark COCO validation set (FID)
Metrics: relationship generation accuracy object generation accuracy (OGA) FID user preference rate
Baselines: Stable Diffusion Personalized Diffusion
Builds on: Stable Diffusion CLIP Graph Convolutional Network Personalized diffusion (Textual Inversion/DreamBooth-style text encoder tuning)
Closest papers by meaning
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models (NeurIPS 2024 · similarity 0.88)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.88)
- RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
- DreamRelation: Bridging Customization and Relation Generation (CVPR 2025 · similarity 0.87)
- Interpretable Diffusion via Information Decomposition (ICLR 2024 · similarity 0.87)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.