Research map / Graph learning / Knowledge and temporal graphs
Scene Graph-Grounded Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGG-IG, which generates images from scene graphs without intermediate layouts by pre-training a masked auto-encoder/relation-embedding scene graph encoder with a spatial contrastive loss, then fine-tuning Stable Diffusion with an image-scene alignment loss.
Problem
Scene graph-to-image generation methods that predict an intermediate layout lose non-spatial interaction relations (e.g., standing on, wearing), and end-to-end diffusion fine-tuning on scene graph embeddings struggles to align them with the text-to-image control, causing object distortions or missin
Approach
Learn a global scene graph embedding via masked node reconstruction and a contrastive spatial constraint against images, then align this embedding with noisy image latents at every denoising timestep (image-scene alignment loss) while fine-tuning Stable Diffusion, removing the need for an intermediate layout.
Key results
- On COCO-Stuff at 128x128, SGG-IG reaches FID 26.2 and IS 19.3 versus 30.2 / 14.6 for SGDiff, the strongest baseline.
- On Visual Genome at 128x128, SGG-IG reaches FID 16.3 versus 20.1 for SGDiff (IS 15.2 vs 11.4).
- Ablation shows removing the image-scene alignment loss causes the largest performance drop, while the spatial constraint contributes least.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add image-scene alignment loss during denoising (ablation) | SGG-IG · COCO-Stuff and Visual Genome | FID and IS | Helped |
| Add masked auto-encoder module (ablation) | SGG-IG · COCO-Stuff and Visual Genome | FID and IS | Helped |
| Add spatial constraint loss (ablation) | SGG-IG · COCO-Stuff and Visual Genome | FID and IS | Helped |
Surprising findings
- The spatial contrastive constraint contributes only a minor gain relative to the other components.
Limitations
- Three-stage pipeline (pre-training, fine-tuning, inference) is not end-to-end, as acknowledged by the authors.
- Evaluation limited to FID and IS on two datasets; no reported relation-accuracy metric.
- Qualitative comparisons are on a small set of examples.
Experimental setup
Datasets: Visual Genome COCO-Stuff
Benchmarks: Visual Genome COCO-Stuff
Metrics: FID Inception Score
Baselines: SG2IM PasteGAN WSGC SGDiff Stable Diffusion
Builds on: Stable Diffusion SGDiff InfoNCE masked auto-encoder DDIM
Closest papers by meaning
- Scene Graph Generation by Iterative Message Passing (CVPR 2017 · similarity 0.92)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.90)
- Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation (NeurIPS 2024 · similarity 0.90)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.89)
- Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency (ICML 2024 · similarity 0.89)
- EGTR: Extracting Graph from Transformer for Scene Graph Generation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.