Research map / Image generation / Autoregressive and latent generation
Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RALF, an autoregressive content-aware layout generator that retrieves nearest-neighbor layouts by canvas image similarity (DreamSim) and fuses them via cross-attention, improving layout quality on CGL and PKU.
Problem
Content-aware graphic layout generation suffers from limited training data for the high-dimensional structured layout domain, leading to artifacts such as misaligned underlays and overlapping text.
Approach
Augment an autoregressive layout transformer with retrieval: nearest-neighbor layouts retrieved by image similarity of canvases are encoded by a frozen self-supervised layout encoder and fused with the canvas feature via cross-attention, so the generator can reference real examples.
Key results
- On the PKU annotated test split, RALF achieves FID 3.45 versus 13.59 for the autoregressive baseline and 11.80 for DS-GAN.
- On CGL, RALF reaches FID 1.32 versus 2.89 for the autoregressive baseline, and underlay effectiveness 0.98 vs 0.92 on the unannotated split.
- RALF trained on 3,000 PKU samples outperforms the autoregressive baseline trained on all 7,734 samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add retrieval augmentation to the autoregressive baseline | Autoreg Baseline · PKU annotated test | FID: 13.59 -> 3.45 | Helped |
| add retrieval augmentation to the autoregressive baseline | Autoreg Baseline · PKU annotated test | Und: 0.43 -> 0.92 | Helped |
| add retrieval augmentation to the autoregressive baseline | Autoreg Baseline · CGL annotated test | FID: 2.89 -> 1.32 | Helped |
| add retrieval augmentation to the autoregressive baseline | Autoreg Baseline · CGL annotated test | Occ: 0.125 -> 0.125 | No clear effect |
| add retrieved-layout feature fL and input-image feature fI alongside fC (ablation) | RALF · PKU test | FID: 4.67 -> 3.45 | Helped |
| add cross-attended feature fC (ablation) | RALF · PKU test | FID: 4.08 -> 3.45 | Helped |
| add retrieved-image feature fI (ablation) | RALF · PKU test | FID: 3.45 -> 8.82 | Hurt |
| move retrieval augmentation before the Transformer encoder (ablation) | RALF · PKU test | Und: 0.92 -> 0.72 | Hurt |
| move retrieval augmentation before the Transformer encoder (ablation) | RALF · PKU test | FID: 3.45 -> 2.34 | Helped |
| use DreamSim rather than LPIPS, CLIP, saliency, or random retrieval (ablation) | RALF · PKU test | FID | Helped |
| increase retrieval size K (ablation) | RALF · PKU test | FID | Helped |
| add retrieval augmentation to CGL-GAN | CGL-GAN · PKU test | FID: 34.51 -> 13.28 | Helped |
Surprising findings
- Top-1 nearest-neighbor retrieval alone achieves a strong FID (0.93 on CGL), showing graphic metrics can be fooled by real examples.
- Adding retrieved canvas image features (fI-tilde) degrades performance (FID 8.82 vs 3.45).
Limitations
- Content metrics assume well-designed layouts avoid salient regions, which may not hold in counterexamples.
- Layout encoder depends on a fixed number of element categories; open-vocabulary extension is needed.
- FID can be easily fooled by real examples, as Top-1 Retrieval shows.
Experimental setup
Datasets: CGL PKU
Benchmarks: CGL PKU
Metrics: FID Underlay effectiveness Overlay Occlusion Readability score
Baselines: CGL-GAN DS-GAN ICVT LayoutDM Autoreg Baseline Top-1 Retrieval
Builds on: CGL-GAN ICVT REALM RDM LayoutDM
Closest papers by meaning
- AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models (CVPR 2025 · similarity 0.88)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.85)
- Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints (ICLR 2024 · similarity 0.85)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.85)
- Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation (CVPR 2024 · similarity 0.84)
- Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.