Research map / Vision-language and multimodal / Multimodal LLMs
SKE-Layout: Spatial Knowledge Enhanced Layout Generation with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SKE-Layout, an LLM-based layout generation framework that retrieves spatial knowledge from real and synthetic data via contrastive and multitask-trained BERT embeddings, evaluated on robotic object rearrangement and text-to-image layout.
Problem
LLM-based layout generation from text suffers from hallucination, limited and non-diverse layout examples, poor handling of fine-grained spatial details (angles, distances), and retrieval that overemphasizes object identity over spatial relations.
Approach
Builds a mixed spatial knowledge base (VLM/DETR-extracted real-world layouts plus LLM-generated synthetic tasks with mathematical constraints) and trains a task-specific BERT retrieval embedding with instance-wise contrastive learning on object-anonymized augmentations plus multitask learning, to inject the most relevant spatial knowledge into LLM prompts.
Key results
- On NSR-1K spatial reasoning, SKE-Layout reaches 95.05% layout accuracy and 61.48% GLIP accuracy, outperforming LayoutGPT (GPT-4) by about 3% and 1%.
- On SK-Dataset complex tasks with GPT-4, success rate rises from 4.5% (LLM-GROP) to 21.7%.
- On the StructFormer dataset, improvements over StructFormer are 30%, 20%, 20%, and 10% for Tower, Line, Circle, and Set the Table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SKE-Layout knowledge retrieval and layout planning | GPT-4 · SK-Dataset simple tasks | success rate (%): 69.4 -> 75.1 | Helped |
| add SKE-Layout knowledge retrieval and layout planning | GPT-4 · SK-Dataset complex tasks | success rate (%): 4.5 -> 21.7 | Helped |
| add contrastive learning (ablation) | SKE-Layout with GPT-4 · SK-Dataset simple tasks | success rate (%): 71.5 -> 75.1 | Helped |
| add contrastive learning (ablation) | SKE-Layout with GPT-4 · SK-Dataset complex tasks | success rate (%): 19.2 -> 21.7 | Helped |
| add contrastive learning (ablation) | SKE-Layout · NSR-1K numerical reasoning | accuracy: 95.28 -> 97.24 | Helped |
| add contrastive learning (ablation) | SKE-Layout · NSR-1K spatial reasoning layout eval. | accuracy: 56.86 -> 59.70 | Helped |
| add multitask learning (ablation) | SKE-Layout · NSR-1K numerical reasoning | accuracy: 97.11 -> 97.24 | Helped |
| add multitask learning (ablation) | SKE-Layout · NSR-1K spatial reasoning layout eval. | accuracy: 58.68 -> 59.70 | Helped |
| use SKE-Layout instead of LayoutGPT | GPT-4 text-to-layout-to-image pipeline · NSR-1K numerical reasoning | precision: 78.36 -> 85.28 | Mixed |
| use SKE-Layout instead of LayoutGPT | GPT-4 text-to-layout-to-image pipeline · NSR-1K numerical reasoning | accuracy: 78.43 -> 97.24 | Helped |
| use SKE-Layout instead of LayoutGPT | GPT-4 text-to-layout-to-image pipeline · NSR-1K spatial reasoning layout eval. | accuracy: 55.64 -> 59.70 | Helped |
| use SKE-Layout instead of LayoutGPT | GPT-4 text-to-layout-to-image pipeline · NSR-1K spatial reasoning image eval. | accuracy (GLIP): 60.64 -> 62.19 | Helped |
Surprising findings
- Contrastive and multitask learning yield minimal differences on the StructFormer dataset, attributed to its simpler spatial relations.
Limitations
- Open-domain scenarios with unknown object sets are not addressed.
- Complex task success rates remain low (21.7% at best on SK-Dataset with GPT-4).
- Synthetic data validation relies partly on human evaluators, which limits scalability.
Experimental setup
Datasets: NSR-1K StructFormer Dataset SK-Dataset instructions from prior work (LLM-GROP, others)
Benchmarks: SK-Dataset StructFormer Dataset NSR-1K
Metrics: success rate precision recall layout accuracy GLIP accuracy
Baselines: StructFormer LLM-GROP Stable Diffusion Attend-and-Excite LayoutTransformer LayoutGPT
Builds on: LayoutGPT SimCSE RAG LLM-GROP GLIGEN
Closest papers by meaning
- LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.88)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.88)
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.87)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.87)
- Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.