Research map / Image generation / Autoregressive and latent generation
Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Layout Composer, a dual image-vector diffusion model that jointly denoises a Stable Diffusion image layout and per-element bounding boxes, exchanging cross-attention maps between the two branches.
Problem
Automatic design layout generation mostly ignores element-level visual information (image content, typeface, color), producing layouts that are geometrically plausible but visually weak, and existing visual layout datasets are small.
Approach
Couple a pre-trained Stable Diffusion v1.5 image diffusion model with a transformer-based vector diffusion model over bounding boxes, exchanging processed U-Net cross-attention maps (with attention modulation and a localization loss) as the shared medium, and mask latents outside the canvas to handle aspect ratio.
Key results
- On RICO, VLC (Vector only) reaches FID 2.38 and mIoU 0.418 versus LayoutDM's 3.55 and 0.277.
- On PubLayNet, VLC (Vector only) reports FID 5.49 vs LayoutDM 7.95; dual-domain gets 5.74.
- On Poster, dual-domain bounding-box FID is 0.09 vs LayoutDM 1.81, and generated-image FID is 6.00.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the image diffusion domain to vector-only VLC (ablation) | VLC · Crello | composed image FID: 5.74 -> 5.83 | Hurt |
| Add the image diffusion domain to vector-only VLC (ablation) | VLC · Crello | bounding box FID: 0.21 -> 0.21 | No clear effect |
| Add the image diffusion domain to vector-only VLC (ablation) | VLC · Poster | composed image FID: 5.82 -> 3.75 | Helped |
| Add the image diffusion domain to vector-only VLC (ablation) | VLC · Poster | bounding box FID: 0.19 -> 0.09 | Helped |
| Use image embeddings and visual input in place of category-only LayoutDM conditioning | VLC (Vector only) · Crello | composed image FID: 34.43 -> 5.74 | Helped |
| Use image embeddings and visual input in place of category-only LayoutDM conditioning | VLC (Vector only) · Poster | composed image FID: 20.00 -> 5.82 | Helped |
| Use vector-only VLC instead of LayoutDM | RICO · RICO | FID: 3.55 -> 2.38 | Helped |
| Use vector-only VLC instead of LayoutDM | PubLayNet · PubLayNet | FID: 7.95 -> 5.49 | Helped |
| Use dual-domain VLC instead of LayoutDM | VLC (Dual-domain) · RICO | FID: 3.55 -> 5.60 | Hurt |
| Use dual-domain VLC instead of LayoutDM | VLC (Dual-domain) · PubLayNet | FID: 7.95 -> 5.74 | Helped |
Surprising findings
- On RICO, the dual-domain model (FID 5.60) is worse than the vector-only variant (FID 2.38) on bounding-box FID.
- Fine-tuning the image diffusion model made it favor square layouts, ignoring the specified canvas aspect ratio, which required canvas masking.
Limitations
- Vector-only and dual-domain models perform similarly on Crello, attributed to many elements causing image-vector content mismatch.
- FID is biased toward image quality and content similarity rather than layout quality.
- Method is slower than baselines due to computation.
Experimental setup
Datasets: Poster (new, 137,781 layouts) Crello RICO PubLayNet
Benchmarks: Crello Poster RICO PubLayNet
Metrics: FID maximum IoU
Baselines: LayoutDM FlexDM LayoutVAE NDN-none LayoutGAN++ LayoutTrans MaskGIT BLT BART VQDiffusion
Builds on: Stable Diffusion DDPM LayoutDM ControlNet Attention Modulation
Closest papers by meaning
- Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints (ICLR 2024 · similarity 0.89)
- DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models (CVPR 2025 · similarity 0.87)
- HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation (NeurIPS 2024 · similarity 0.87)
- MaskPLAN: Masked Generative Layout Planning from Partial Input (CVPR 2024 · similarity 0.86)
- PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation (CVPR 2025 · similarity 0.86)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.