Research map / Image generation / Text-to-image diffusion
Compositional Text-to-Image Generation with Dense Blob Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BlobGEN, a Stable Diffusion-based text-to-image model grounded on dense blob representations (tilted ellipses plus region captions) via a masked cross-attention module, and uses in-context LLM prompting to generate blobs for compositional generation.
Problem
Text-to-image diffusion models struggle to follow complex prompts and offer limited fine-grained layout controllability; bounding boxes are coarse and dense maps are hard to construct.
Approach
Decompose scenes into dense blob representations (tilted ellipse parameters plus a region-level text description), inject them via a masked cross-attention so each blob's description only attends to its local region, and use in-context learning with LLMs (GPT-3.5/GPT-4 for parameters, LLaMA-13B for descriptions) to generate blobs from prompts.
Key results
- Zero-shot FID on MS-COCO improves from 10.40 (base SD-1.4 with consistency decoder) to 8.61, and mIOU rises from 0.4238 (GLIGEN) to 0.5103.
- On NSR-1K spatial reasoning with GPT-4 layouts, image accuracy is 80.16% vs 74.35% for LayoutGPT (GPT4).
- On NSR-1K numerical reasoning with GPT-4, image accuracy is 62.96% vs 61.54% for LayoutGPT (GPT3.5-chat).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use blob representations with BlobGEN instead of bounding-box grounding with GLIGEN | SD-1.4-based text-to-image generation · MS-COCO validation, SD decoder | FID: 11.63 -> 8.94 | Helped |
| Use blob representations with BlobGEN instead of bounding-box grounding with GLIGEN | SD-1.4-based text-to-image generation · MS-COCO validation, consistency decoder | mIOU: 0.4238 -> 0.5103 | Helped |
| Generate blob layouts rather than box layouts | GPT4-assisted image generation · NSR-1K numerical reasoning | Image Acc: 60.25 -> 62.96 | Helped |
| Generate blob layouts rather than box layouts | GPT4-assisted image generation · NSR-1K spatial reasoning | Image Acc: 74.35 -> 80.16 | Helped |
| Add blob parameter embeddings to blob text embeddings through concatenation (ablation) | BlobGEN · MS-COCO validation, ablation evaluation | FID: 9.21 -> 9.14 | Helped |
| Add local-region masking in cross-attention (ablation) | BlobGEN · MS-COCO validation, ablation evaluation | mIOU: 0.4401 -> 0.4805 | Helped |
| Add the masked cross-attention module (ablation) | BlobGEN · MS-COCO validation, ablation evaluation | FID: 9.47 -> 9.14 | Helped |
| Add gated self-attention (ablation) | BlobGEN · MS-COCO validation, ablation evaluation | FID: 9.79 -> 9.14 | Helped |
| Add gated self-attention (ablation) | BlobGEN · MS-COCO validation, ablation evaluation | mIOU: 0.4898 -> 0.4805 | Mixed |
| Use an object-focused prompt to generate blob descriptions (ablation) | BlobGEN training data · MS-COCO validation, ablation evaluation | FID: 9.65 -> 9.14 | Helped |
Surprising findings
- Retrieval of in-context demonstrations is not necessary: fixed in-context examples achieve close performance (spatial image accuracy 77.84% vs 80.16%).
Limitations
- Blob representations cannot perfectly reconstruct original images without inversion methods.
- Editing failures occur when background blobs overlap the edited object or edits change unrelated content.
- Sensitivity to LLM-generated blobs can cause copy-and-paste artifacts or extra objects.
Experimental setup
Datasets: Common Crawl image-text pairs (1M subset) MS-COCO NSR-1K
Benchmarks: MS-COCO validation NSR-1K
Metrics: FID mIOU rCLIP_t rCLIP_i layout precision layout recall layout accuracy image accuracy
Baselines: GLIGEN SD-1.4 LayoutGPT LMD SDXL Attend-and-Excite
Builds on: Stable Diffusion GLIGEN LayoutGPT ODISE LLaVA-1.5
Closest papers by meaning
- BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations (CVPR 2025 · similarity 0.92)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.89)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.88)
- ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty (NeurIPS 2024 · similarity 0.86)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.86)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.