Research map / Image generation / Autoregressive and latent generation
Desigen: A Pipeline for Controllable Design Template Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Desigen is a pipeline that generates design template backgrounds with saliency-constrained diffusion and attention reduction, then synthesizes layouts with a Transformer decoder, refined iteratively, trained on a 40k banner dataset.
Problem
Automating design template creation requires backgrounds that preserve non-salient space for overlaying layout elements, plus harmonious layouts, which off-the-shelf text-to-image models do not provide with spatial control.
Approach
Adds a cross-attention saliency constraint loss during diffusion fine-tuning and a training-free attention reduction over user-given masks to keep space free for layout; a background-conditioned autoregressive layout Transformer is then iteratively refined with the background.
Key results
- Salient ratio reduced from 35.92 (Stable Diffusion) to 20.65 (Desigen), with FID improving from 39.36 to 31.52.
- Layout generator occlusion 13.47 vs 13.53 (DS-GAN) and 28.26 (LayoutTransformer); alignment 0.35 and overlap 14.41, close to real data (0.33, 11.32).
- Iterative refinement over three rounds lowers occlusion from 11.66 to 8.69 and salient ratio from 16.87 to 15.27.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the saliency map mask instead of a full mask for the attention constraint (ablation) | Desigen background generator · Web-design test set | FID: 34.49 -> 31.52 | Helped |
| Use the saliency map mask instead of a full mask for the attention constraint (ablation) | Desigen background generator · Web-design test set | Salient Ratio: 21.25 -> 20.65 | Helped |
| Fine-tune Stable Diffusion on Web-design | Stable Diffusion · Web-design test set | Salient Ratio: 35.92 -> 23.61 | Helped |
| Apply Desigen's background-generation method instead of Stable Diffusion (FT) | Stable Diffusion · Web-design test set | FID: 37.41 -> 31.52 | Helped |
| Apply Desigen's background-generation method instead of Stable Diffusion (FT) | Stable Diffusion · Web-design test set | CLIP Score: 29.53 -> 29.20 | Hurt |
| Apply Desigen's background-generation method instead of Stable Diffusion (FT) | Stable Diffusion · Human assessment | Cleanliness: 3.45 -> 3.95 | Helped |
| Apply Desigen's background-generation method instead of Stable Diffusion (FT) | Stable Diffusion · Human assessment | Aesthetics: 3.36 -> 3.38 | Helped |
| Apply Desigen's background-generation method instead of Stable Diffusion (FT) | Stable Diffusion · Human assessment | Relevancy: 3.95 -> 3.99 | Helped |
| Use the Swin image encoder instead of ResNet for background conditioning (ablation) | Desigen layout generator · Layout generation test set | Occlusion: 21.44 -> 13.47 | Helped |
| Use the Swin image encoder instead of ViT for background conditioning (ablation) | Desigen layout generator · Layout generation test set | Occlusion: 18.13 -> 13.47 | Helped |
| Perform iterative background and layout refinement (ablation) | Desigen · Design refinement | Occlusion: 11.66 -> 8.96 | Helped |
| Perform iterative background and layout refinement (ablation) | Desigen · Design refinement | Salient Ratio: 16.87 -> 15.66 | Helped |
Surprising findings
- Full-mask attention suppression (Mx=1) performs worse than the saliency-map constraint due to convergence difficulty.
Limitations
- Baselines are limited since no prior text-to-design method exists for direct comparison.
- Larger attention-reduction mask ratios force near-empty backgrounds and degrade FID.
- CLIP score is lower than Stable Diffusion, reflecting a trade-off between text relevance and free space.
Experimental setup
Datasets: Web-design
Benchmarks: Web-design
Metrics: salient ratio FID CLIP score alignment overlap occlusion human assessment (1-5)
Baselines: Stable Diffusion DALL-E 2 Stable Diffusion (FT) LayoutTransformer CanvasVAE DS-GAN
Builds on: Stable Diffusion LayoutTransformer prompt-to-prompt Paint-with-words
Closest papers by meaning
- Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints (ICLR 2024 · similarity 0.87)
- Graphic Design with Large Multimodal Model (AAAI 2025 · similarity 0.86)
- AutoPresent: Designing Structured Visuals from Scratch (CVPR 2025 · similarity 0.85)
- Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation (CVPR 2024 · similarity 0.85)
- DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models (CVPR 2025 · similarity 0.84)
- Spot the Error: Non-autoregressive Graphic Layout Generation with Wireframe Locator (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.