Research map / Image generation / Text-to-image diffusion
MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MuLAn, a 44.8K-image dataset of multi-layer RGBA instance decompositions of COCO and LAION images, built with a training-free pipeline of pretrained detection, segmentation, depth, inpainting and matting models.
Problem
Text-to-image generation lacks precise spatial controllability and instance-level compositionality because outputs are flat RGB images, and there is no photorealistic public dataset with multi-layer instance and occlusion annotations.
Approach
A training-free pipeline decomposes a single RGB image into an ordered stack of RGBA instance layers plus background, using open-vocabulary detection, SAM segmentation, MiDaS depth, occlusion graphs, iterative diffusion inpainting of occluded areas, and ViTMatte alpha refinement; the outputs are curated into a dataset.
Key results
- Pipeline success rate of 36% on 5000 manually annotated LAION images (52% counting minor defects), leading to 44.8K curated decompositions (16K COCO, 28.9K LAION).
- MuLAn covers 759 categories across 101,269 instances, with 7.7% average occlusion rate.
- In instance addition, the MuLAn-finetuned InstructPix2Pix shows better and more consistent detection and scene preservation than the original InstructPix2Pix on EditVal and the attribute test set, with lower attribute bleeding (qualitative and quantitative).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Stable Diffusion v1.5 on MuLAn instances for RGBA image generation | Stable Diffusion v1.5 · RGBA image generation | Generated RGBA instance quality | Helped |
| Fine-tune InstructPix2Pix using MuLAn layer-addition training triplets | InstructPix2Pix · EditVal instance addition and Attribute Test Set | Scene preservation | Helped |
Surprising findings
- Only 36% of automatically decomposed images were judged successful by manual inspection, with segmentation the biggest failure mode.
Limitations
- Low pipeline success rate (36%) with segmentation, inpainting and detection failures
- Relies on automatic captions and generic inpainting prompts
- Occlusion-completion quality bounded by inpainting models and mask heuristics
Experimental setup
Datasets: COCO LAION Aesthetics 6.5 THINGS WordNet EditVal 15,791 matting-dataset instances
Benchmarks: EditVal instance addition Attribute test set (authors' own)
Metrics: OWL-ViT 2 detection confidence BLIP2 VQA detection rate L2 distance between original and edited images manual failure-mode distribution
Baselines: InstructPix2Pix Stable Diffusion v1.5 (matting-data finetune) Text2Layer PCNet
Builds on: SAM detCLIPv2 MiDaS InstaDepth Stable Diffusion InstructPix2Pix Text2Layer
Closest papers by meaning
- Generating compositional scenes via Text-to-image RGBA Instance Generation (NeurIPS 2024 · similarity 0.88)
- PixelART: Image-to-Layer Decomposition without Latents or Text-to-Image Pretraining (NeurIPS 2026 · similarity 0.87)
- LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration (NeurIPS 2025 · similarity 0.85)
- Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis (CVPR 2024 · similarity 0.85)
- MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost (ICML 2025 · similarity 0.84)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.