Research map / Graph learning / Graph self-supervised learning
InstructG2I: Synthesizing Images from Multimodal Attributed Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Graph2Image (generating node images from multimodal attributed graphs) and InstructG2I, a Stable Diffusion-based model that conditions on sampled graph neighbors via semantic PPR sampling and a Graph-QFormer, with graph classifier-free guidance.
Problem
Generating images for a node in a multimodal attributed graph, conditioned on the node's text and its graph neighborhood, which is hard due to graph size explosion, entity dependencies, and the need for controllability over graph conditions.
Approach
Compress graph neighbors into graph prompt tokens via semantic personalized-PageRank sampling (structure then CLIP-based reranking) and a Graph-QFormer that uses self-attention for image-image dependencies and text-guided cross-attention for text-image dependencies; add separate guidance scales per graph edge at inference.
Key results
- InstructG2I reaches CLIP/DINOv2 scores of 73.73/46.45 on ART500K, 68.34/51.70 on Amazon, and 50.37/25.54 on Goodreads, beating all baselines including SD-1.5 FT (66.55/34.65 on ART500K).
- Ablation shows removing Graph-QFormer drops ART500K DINOv2 from 46.45 to 44.16, and GraphSAGE/GAT variants score lower still.
- InstructPix2Pix obtains a better FID than InstructG2I, which the authors attribute to its in-distribution input image.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| condition generation on graph information using InstructG2I instead of the strongest baseline | Image generation · ART500K | DINOv2 score: 34.65 -> 46.45 | Helped |
| condition generation on graph information using InstructG2I instead of the strongest baseline | Image generation · Amazon | DINOv2 score: 41.52 -> 51.70 | Helped |
| condition generation on graph information using InstructG2I instead of the strongest baseline | Image generation · Goodreads | DINOv2 score: 20.94 -> 25.54 | Helped |
| add Graph-QFormer instead of simple image-encoder features (ablation) | InstructG2I · ART500K | DINOv2 score: 44.16 -> 46.45 | Helped |
| add Graph-QFormer instead of simple image-encoder features (ablation) | InstructG2I · Amazon | DINOv2 score: 48.18 -> 51.70 | Helped |
| add Graph-QFormer instead of simple image-encoder features (ablation) | InstructG2I · Goodreads | DINOv2 score: 24.74 -> 25.54 | Helped |
| use Graph-QFormer rather than GraphSAGE (ablation) | InstructG2I · Amazon | DINOv2 score: 43.40 -> 51.70 | Helped |
| use Graph-QFormer rather than GAT (ablation) | InstructG2I · Goodreads | DINOv2 score: 21.45 -> 25.54 | Helped |
| use graph prompts rather than neighbor images as direct IP2P input (ablation) | InstructG2I · ART500K | DINOv2 score: 33.90 -> 46.45 | Helped |
| use graph prompts rather than neighbor images as direct IP2P input (ablation) | InstructG2I · Amazon | DINOv2 score: 40.32 -> 51.70 | Helped |
| use graph prompts rather than neighbor images as direct IP2P input (ablation) | InstructG2I · Goodreads | DINOv2 score: 21.55 -> 25.54 | Helped |
| use neighbor images rather than neighbor texts for graph conditioning (ablation) | InstructG2I · ART500K | DINOv2 score: 38.64 -> 46.45 | Helped |
Surprising findings
- InstructPix2Pix achieves a better FID than InstructG2I despite lower instance-level similarity scores.
Limitations
- Uses Stable Diffusion 1.5 only; larger models such as SDXL are left for future work.
- Models the graph as homogeneous, ignoring heterogeneous node and edge types.
- Inherits Stable Diffusion's known risks such as NSFW content and adversarial vulnerability.
Experimental setup
Datasets: ART500K Amazon Goodreads
Benchmarks: ART500K Amazon Goodreads
Metrics: CLIP score DINOv2 score FID
Baselines: SD-1.5 SD-1.5 FT InstructPix2Pix ControlNet GraphSAGE GAT
Builds on: Stable Diffusion classifier-free guidance Q-Former personalized PageRank InstructPix2Pix
Closest papers by meaning
- FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts (CVPR 2025 · similarity 0.88)
- Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning (CVPR 2025 · similarity 0.87)
- R3CD: Scene Graph to Image Generation with Relation-Aware Compositional Contrastive Control Diffusion (AAAI 2024 · similarity 0.87)
- Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation (NeurIPS 2024 · similarity 0.87)
- GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs (CVPR 2025 · similarity 0.86)
- Editing Partially Observable Networks via Graph Diffusion Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.