Research map / Image generation / Autoregressive and latent generation
R3CD: Scene Graph to Image Generation with Relation-Aware Compositional Contrastive Control Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R3CD, a scene-graph transformer (SGFormer, T5-initialized) plus joint attention-map and diffusion-step contrastive losses with triplet-level compositional generation, to improve abstract-relation and entity fidelity in scene-graph-to-image diffusion.
Problem
Scene graph to image generation with diffusion models struggles to depict abstract (non-positional) relations between entities and often omits entities in complex scene graphs.
Approach
Encodes nodes and edges with a graph transformer initialized by T5 embeddings, then applies contrastive losses on cross-attention maps and on predicted noise across early denoising steps (positives share the same abstract relation), and generates images triplet by triplet via edge-embedding attention maps.
Key results
- R3CD (SGFormer+R3CD) achieves IS 19.5 on COCO-Stuff and 18.9 on Visual Genome, versus 17.8 and 16.4 for SGDiff.
- FID improves from 36.2 (SGDiff) to 32.9 on COCO-Stuff and from 26.0 to 23.4 on Visual Genome.
- On relation synthesis, G2I-ACC rises from 70.1 (object-only GCN) to 73.8 for the full R3CD model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GCN with SGFormer (ablation) | R3CD · COCO-Stuff | IS: 16.5 -> 17.5 | Helped |
| replace GCN with SGFormer (ablation) | R3CD · COCO-Stuff | FID: 38.6 -> 35.7 | Helped |
| add R3CD contrastive control to SGFormer (ablation) | R3CD · COCO-Stuff | IS: 17.5 -> 19.5 | Helped |
| add R3CD contrastive control to SGFormer (ablation) | R3CD · Visual Genome | FID: 25.2 -> 23.4 | Helped |
| add relations to object-only GCN (ablation) | Relation synthesis · Relation synthesis | G2I-ACC: 70.1 -> 72.4 (+2.3) | Helped |
| replace GCN with SGFormer (ablation) | Obj + Rel · Relation synthesis | G2I-ACC: 72.4 -> 72.9 (+2.8) | Helped |
| add compositional generation (ablation) | SGFormer · Relation synthesis | G2I-ACC: 72.9 -> 73.2(+3.1) | Helped |
| add R3CD contrastive control (ablation) | SGFormer · Relation synthesis | G2I-ACC: 73.2 -> 73.8(+3.7) | Helped |
| replace GCN Diffusion with SGFormer (ablation) | Image generation · COCO-Stuff | IS: 16.0 -> 17.3 | Helped |
| add attention map contrastive loss (ablation) | SGFormer · COCO-Stuff | FID: 25.2 -> 24.6 | Helped |
| add diffusion steps contrastive loss (ablation) | SGFormer · COCO-Stuff | FID: 25.2 -> 24.0 | Helped |
| add both attention map and diffusion steps contrastive losses (ablation) | SGFormer · COCO-Stuff | IS: 17.3 -> 18.9 | Helped |
Limitations
- Relation accuracy gains over the GCN baseline are modest (about 1-4 points).
- Hyperparameters for the contrastive losses and the number of early denoising steps are set without a reported sensitivity analysis.
Experimental setup
Datasets: Visual Genome COCO-Stuff
Benchmarks: Visual Genome COCO-Stuff
Metrics: IS FID
Baselines: Sg2Im WSGC SOAP PasteGAN KCGM SGDiff
Builds on: SGDiff Attention-based semantic guidance (Feng et al. 2022) Prompt-to-prompt attention maps (Hertz et al. 2022) T5
Closest papers by meaning
- Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation (NeurIPS 2024 · similarity 0.91)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.89)
- GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs (CVPR 2024 · similarity 0.89)
- FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts (CVPR 2025 · similarity 0.88)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.88)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.