Research map / Large language models / Language, safety and interpretability
LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Recasts graphic layout generation as HTML code generation, using an LLM (CodeLLaMA/LLaMA2 7B) fine-tuned with Code Instruct Tuning to fill masked code, then rendering it into layouts.
Problem
Existing layout generation methods treat layouts as numerical tuples, ignoring semantic relationships between elements; the paper asks whether LLMs can generate layouts via code to add semantics and exploit LLM knowledge.
Approach
Represent layouts as HTML/SVG code with masked numerical values (Code Initialization), have an LLM complete the masks via instruction-tuned code completion with permutation-based self-consistency and multi-task joint loss, then render the code directly to a layout.
Key results
- On Magazine, FID improves from 19.206 (LayoutDM) to 8.791 for LayoutNUWA (CodeLLaMA, domain-agnostic) in the C→S+P task, and the paper claims at least 50% FID gains over the best baseline on the low-resource Magazine dataset.
- On the Magazine completion task, LayoutNUWA reaches mIoU 0.495 vs 0.328 for LayoutDM.
- Zero-shot LLaMA2 and CodeLLaMA fail on 100% of generations, while GPT-4 zero-shot fails on about 28-34%; LayoutNUWA has 0-0.3% failure after tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate layouts as code rather than numerical output (ablation) | LayoutNUWA-L2-DS · Magazine, C+S → P | mIoU: 0.000 -> 0.358 | Helped |
| Generate layouts as code rather than numerical output (ablation) | LayoutNUWA-L2-DS · Magazine, C+S → P | FID: 349.231 -> 4.682 | Helped |
| Add the HTML code template (ablation) | LayoutNUWA-L2-DS · Magazine, C → S+P | FID: 16.324 -> 9.741 | Helped |
| Add the HTML code template (ablation) | LayoutNUWA-L2-DS · Magazine, C+S → P | FID: 12.432 -> 4.682 | Helped |
| Add the HTML code template (ablation) | LayoutNUWA-L2-DS · Magazine, Completion | FID: 15.732 -> 7.257 | Helped |
| Add task instructions and code template rather than numerical tuning (ablation) | LayoutNUWA-L2-DS · Magazine, C → S+P | FID: 17.982 -> 9.741 | Helped |
| Add task instructions and code template rather than numerical tuning (ablation) | LayoutNUWA-L2-DS · Magazine, C+S → P | FID: 14.326 -> 4.682 | Helped |
| Add task instructions and code template rather than numerical tuning (ablation) | LayoutNUWA-L2-DS · Magazine, Completion | FID: 16.243 -> 7.257 | Helped |
| Use domain-agnostic rather than domain-specific training | LayoutNUWA-L2 · Magazine, C → S+P | FID: 9.741 -> 9.632 | Helped |
| Use CodeLLaMA rather than LLaMA2 | LayoutNUWA, Magazine · C → S+P | FID: 9.741 -> 8.985 | Helped |
| Use CodeLLaMA with domain-agnostic training rather than LLaMA2 with domain-specific training | LayoutNUWA · Magazine, C → S+P | FID: 9.741 -> 8.791 | Helped |
| Use LayoutNUWA code generation rather than LayoutDM | LayoutNUWA · Magazine, C → S+P | FID: 19.206 -> 8.791 | Helped |
Surprising findings
- Numerical-format output (code infilling of only masked values) yields very high failure rates (up to 78% in C→S+P) and near-zero mIoU, versus code format.
- Zero-shot GPT-4 with the CIT prompt produces layouts whose FID cannot be computed due to large distribution mismatch.
Limitations
- Autoregressive generation is slower than non-autoregressive models.
- Error propagation in AR generation can degrade later elements when training is insufficient.
- Several tasks do not achieve best results on RICO and PubLayNet, and some mIoU/FID trade-offs appear in tables.
Experimental setup
Datasets: RICO PubLayNet Magazine
Benchmarks: RICO PubLayNet Magazine
Metrics: FID mIoU Alignment Overlap Fail ratio
Baselines: LayoutTrans BLT LayoutGAN++ MaskGIT DiffusionLM LayoutDM GPT-4 (zero-shot)
Builds on: LayoutDM Adaptive Quantization LLaMA 2 CodeLLaMA DeepSpeed
Closest papers by meaning
- Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement (CVPR 2026 · similarity 0.88)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence (ICLR 2026 · similarity 0.86)
- AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models (CVPR 2025 · similarity 0.86)
- VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.85)
- LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.