Research map / Image generation / Text-to-image diffusion
SemanticDraw: Towards Real-Time Interactive Content Creation from Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Makes region-based multi-prompt diffusion (MultiDiffusion) compatible with few-step accelerators and adds a multi-prompt stream batch pipeline, enabling sub-second interactive hand-drawn semantic painting on a single RTX 2080 Ti.
Problem
Region-based text-to-image control (MultiDiffusion) is very slow (about 52 s at 512x512) and incompatible with few-step acceleration methods like LCM, which blocks real-time interactive content creation.
Approach
Three stabilization strategies (latent pre-averaging that separates deterministic denoising from noise addition, mask-centering bootstrapping with white background, and quantized Gaussian-blurred masks tied to noise levels) make MultiDiffusion work with accelerated schedulers, plus a multi-prompt stream batch pipeline where each batch element has a different timestep to raise throughput.
Key results
- Stabilized acceleration gives about 9.7x speedup over MultiDiffusion with comparable or better FID (e.g., SD1.5 LCM FID 93.93 vs 70.93 for DDIM baseline reference, and 270.55 for unstabilized MD+LCM).
- Multi-prompt stream batch pipeline plus Tiny AutoEncoder reaches 1.57 FPS (0.64 s per frame) on a single RTX 2080 Ti, an 83.1x throughput gain over the baseline.
- In a user study, 90.9% preferred SemanticDraw outputs over MultiDiffusion and 0% over MultiDiffusion+LCM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add latent pre-averaging (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | FID: 270.55 -> 80.64 | Helped |
| Add latent pre-averaging (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | CLIPfg: 22.53 -> 22.80 | Helped |
| Add latent pre-averaging (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | CLIPbg: 19.63 -> 26.95 | Helped |
| Add mask-centering bootstrapping (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | FID: 80.64 -> 79.54 | Helped |
| Add mask-centering bootstrapping (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | CLIPfg: 22.80 -> 23.06 | Helped |
| Add mask-centering bootstrapping (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | CLIPbg: 26.95 -> 26.72 | Hurt |
| Add quantized masks (σ = 4) (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | FID: 79.54 -> 78.21 | Helped |
| Add quantized masks (σ = 4) (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | CLIPfg: 23.06 -> 23.08 | Helped |
| Add quantized masks (σ = 4) (ablation) | SemanticDraw stabilization · COCO validation, SD1.5 (512 × 512) | CLIPbg: 26.72 -> 26.72 | No clear effect |
| Add stable acceleration (ablation) | SemanticDraw throughput pipeline · 512 × 512 images from three prompt-mask pairs, RTX 2080 Ti | Throughput (FPS): 0.0189 -> 0.183 | Helped |
| Add multi-prompt stream batch (ablation) | SemanticDraw throughput pipeline · 512 × 512 images from three prompt-mask pairs, RTX 2080 Ti | Throughput (FPS): 0.183 -> 1.38 | Helped |
| Add Tiny AutoEncoder (ablation) | SemanticDraw throughput pipeline · 512 × 512 images from three prompt-mask pairs, RTX 2080 Ti | Throughput (FPS): 1.38 -> 1.57 | Helped |
Surprising findings
- Directly combining MultiDiffusion with LCM produces blurry, noisy outputs (FID 270.55), showing the two techniques are incompatible without modification.
Limitations
- Quality depends on mask-fidelity versus harmonization trade-off controlled by the quantized mask blur parameter.
- Text encoding accounts for about 40% of runtime, and prompt changes trigger pipeline flush and reinitialization.
Experimental setup
Datasets: COCO validation
Benchmarks: COCO validation
Metrics: FID Inception Score CLIP score runtime FPS user preference
Baselines: MultiDiffusion MultiDiffusion + LCM DDIM sampler Hyper-SD Euler Discrete Flash Flow Match
Builds on: MultiDiffusion StreamDiffusion Latent Consistency Models Stable Diffusion
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.87)
- ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer (ICLR 2025 · similarity 0.87)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.87)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.87)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.87)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.