Research map / Image generation / Text-to-image diffusion
Synergistic Dual Spatial-aware Generation of Image-to-text and Text-to-image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spatial Dual Discrete Diffusion (SD^3), jointly modeling spatial image-to-text and text-to-image generation through a shared 3D scene graph diffusion and dual learning, evaluated on the VSD dataset.
Problem
Standalone spatial image-to-text (SI2T) and spatial text-to-image (ST2I) models perform poorly at spatial understanding because 3D spatial feature modeling is difficult from 2D images or text.
Approach
Treats SI2T and ST2I as dual tasks sharing a 3D scene graph (3DSG) representation generated by a shared graph discrete diffusion model, and uses intermediate features of the easier 3D-to-X diffusion processes to guide the harder X-to-3D processes.
Key results
- On VSDv1 ST2I, SD^3 reaches FID 11.04 vs. 12.86 for the best baseline Frido; on VSDv2 SI2T, BLEU4 27.63 vs. 26.40 for 3DVSD.
- Removing the 3DSG and dual learning (Singleton) drops VSDv1 SI2T BLEU4 from 56.23 to 48.77.
- Human evaluation: SD^3 spatial accuracy 3.86 for ST2I vs. 3.04 for Frido, and 4.03 for SI2T vs. 3.03 for 3DVSD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3DSG integration (ablation) | Vanilla Dual Learning · VSDv2 | BLEU4: 26.10 -> 27.63 | Helped |
| add dual learning (ablation) | Singleton + 3D · VSDv2 | FID: 10.43 -> 10.09 | Helped |
| add 3DSG modeling and dual learning (ablation) | Singleton · VSDv2 | FID: 14.70 -> 10.09 | Helped |
| add dual intermediate feature sharing (ablation) | 3DSG graph diffusion · 3DSG generation | TriRec.: 65.71 -> 84.51 | Helped |
| add graph diffusion pre-training (ablation) | 3DSG graph diffusion · 3DSG generation | TriRec.: 56.82 -> 84.51 | Helped |
| replace discrete diffusion with continuous diffusion (ablation) | SD³ · VSDv2 | FID: 10.09 -> 10.25 | Hurt |
| replace diffusion SI2T with OFA-based SI2T (ablation) | SD³ · VSDv2 | FID: 10.09 -> 11.20 | Hurt |
| remove DGAE pre-training (ablation) | SD³ training strategy · VSD | ST2I and SI2T performance | Hurt |
| remove spatial alignment (ablation) | SD³ training strategy · VSD | ST2I and SI2T performance | Hurt |
| remove 2DSG-to-3DSG diffusion training (ablation) | SD³ training strategy · VSD | ST2I and SI2T performance | Hurt |
| increase noise in gold 3DSG training data (ablation) | SD³ graph diffusion · VSD | 3DSG and final performance | Hurt |
Surprising findings
- Replacing the diffusion-based SI2T model with a non-diffusion OFA model hurts ST2I (FID 10.09 to 11.20) while SI2T stays comparable, suggesting diffusion SI2T is needed for symmetric dual learning.
- Vanilla dual learning without 3DSG guidance performs only modestly better than a single-task model with 3D features, indicating the 3DSG component carries much of the gain.
Limitations
- Graph diffusion performance depends heavily on the quality of 3DSG training data; corrupting it degrades results sharply.
- Spatial evaluation relies on human judgments from a small set (100 samples, 10 volunteers) and was not fully explored beyond that.
Experimental setup
Datasets: VSD SpatialSense Visual Genome Matterport3D 3DSSG CURB-SG
Benchmarks: VSD
Metrics: FID IS CLIP score BLEU4 SPICE TriRec human Likert spatial accuracy
Baselines: DALLE CogView LAFITE VQ-Diffusion Frido 3DVSD MNIC FNIC DiffCap DDCap SD-1.5 SDXL
Builds on: VQ-Diffusion VQ-GAN dual learning DGAE 3DVSD
Closest papers by meaning
- Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding (NeurIPS 2025 · similarity 0.87)
- Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
- SpatialCLIP: Learning 3D-aware Image Representations from Spatially Discriminative Language (CVPR 2025 · similarity 0.87)
- SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.86)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.