Research map / Image generation / Text-to-image diffusion
InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstanceAssemble, a LoRA-based cascaded Assemble-MMDiT that adds bbox-conditioned, per-instance cropped attention to DiT T2I models for layout-to-image generation, plus the DenseLayout benchmark and Layout Grounding Score metric.
Problem
Layout-to-image generation with precise bounding-box and multimodal instance control remains inaccurate on complex, dense layouts, and existing training-free and training-based methods are slow, costly, or poorly evaluated.
Approach
Process each instance independently by cropping image tokens to its bounding box and running joint attention with its instance tokens, then assemble the updated tokens back via a density-normalized average; apply this as a cascade after the original MMDiT global-prompt pass and train only LoRA adapters plus a layout encoder.
Key results
- On DenseLayout with SD3-Medium, InstanceAssemble reaches mIoU 52.07 vs 15.54 for CreatiLayout and 47.31 for InstanceDiff.
- On LayoutSAM-Eval, the SD3-M variant reports LGS mIoU 78.88 and CropVQA spatial 94.97, the best among compared methods.
- Adds 71M parameters to SD3-Medium (3.46%) and 102M to Flux.1 (0.84%) via rank-4 LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Assemble-Attn (instance-wise cropped attention with assembling) to SD3-Medium (ablation) | SD3-Medium · DenseLayout | mIoU (LGS): 11.69 -> 43.98 | Helped |
| Cascaded design (global prompt-image attention before instance-image attention) instead of parallel (ablation) | Assemble-MMDiT · DenseLayout | mIoU (LGS): 43.98 -> 45.96 | Helped |
| Cascaded design instead of parallel (ablation) | Assemble-MMDiT · DenseLayout | VQA: 84.57 -> 92.71 | Helped |
| Train Assemble-MMDiT with LoRA (rank 4) instead of full fine-tuning (ablation) | Assemble-MMDiT · DenseLayout | mIoU (LGS): 45.96 -> 51.28 | Helped |
| Add DenseSample spatial encoding (ablation) | Assemble-MMDiT · DenseLayout | mIoU (LGS): 51.28 -> 52.07 | Helped |
| Add Assemble-Attn with cascaded LoRA design to SD3-Medium: instance attention vs. base model on layout accuracy (ablation) | SD3-Medium · DenseLayout | VQA: 93.75 -> 93.33 (no DenseSample) / 93.54 (full) | Mixed |
| Replace global-attention-mask region injection with Assemble-Attn on SD3-M (ablation) | SD3-Medium · LayoutSAM-Eval (appendix) | spatial / VQA: spatial 94.11 -> 94.97; VQA 91.53 -> 93.12 | Helped |
| Use visual instance content (image) in addition to text instead of text only (ablation) | InstanceAssemble SD3-M · DenseLayout | mIoU (LGS): 43.72 -> 55.29 | Helped |
| Use depth or edge maps in addition to text instead of text only (ablation) | InstanceAssemble SD3-M · DenseLayout | mIoU (LGS): 43.72 -> 49.64 (depth), 50.73 (edge) | Helped |
| Train on sparse layouts and apply to dense layouts (generalization) | InstanceAssemble SD3-M · DenseLayout | mIoU (LGS): 15.54 (CreatiLayout) -> 52.07 | Helped |
| Use COCO-trained InstanceDiff rather than InstanceAssemble for precise coarse closed-set layouts | COCO-Layout · COCO-Layout | mIoU (LGS): 27.85 (ours) vs 30.39 (InstanceDiff) | Hurt |
| Apply InstanceAssemble to Flux.1-Dev instead of SD3-M backbone | InstanceAssemble · DenseLayout | mIoU (LGS): 43.42 (Flux.1-Dev) vs 52.07 (SD3-M) | Mixed |
Surprising findings
- On COCO-Layout, InstanceDiff (UNet-based) keeps a slightly higher layout mIoU (30.39) than InstanceAssemble (27.85), which the authors attribute to fine-grained COCO training data and entity-wise generation.
- CreatiLayout's LGS mIoU on COCO-Layout is only 7.12, despite strong global quality scores.
Limitations
- Sequential Assemble-MMDiT calls may incur inefficiency; parallelization is future work.
- Image fidelity can degrade in extremely dense or highly complex layouts.
- Slightly lags InstanceDiff in layout precision on COCO-Layout.
Experimental setup
Datasets: LayoutSAM COCO
Benchmarks: LayoutSAM-Eval DenseLayout COCO-Layout
Metrics: Layout Grounding Score (mIoU, color, texture, shape) CropVQA VQA score PickScore CLIPScore
Baselines: InstanceDiff MIGC HICO CreatiLayout Regional-Flux RAG GrounDiT
Builds on: MMDiT GLIGEN flow matching LoRA CreatiLayout
Closest papers by meaning
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.91)
- OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps (NeurIPS 2025 · similarity 0.89)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.89)
- HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation (NeurIPS 2024 · similarity 0.88)
- SpotActor: Training-Free Layout-Controlled Consistent Image Generation (AAAI 2025 · similarity 0.88)
- Control and Realism: Best of Both Worlds in Layout-to-Image without Training (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.