Research map / Image generation / Text-to-image diffusion
Generating compositional scenes via Text-to-image RGBA Instance Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes PixArt-α into an RGBA instance generator with a disentangled VAE and mutual RGB/alpha conditioned sampling, then composes instances layer by layer via multi-layer noise blending for controllable scene generation.
Problem
Layout-controlled text-to-image generation generates all components jointly, limiting fine-grained control over object attributes, relative 3D positioning, and layout editing with content preservation.
Approach
Generate isolated instances natively as RGBA images via a transparency-aware VAE (separate RGB and alpha latent distributions, large KL weight) and a diffusion model trained with paired timestep conditioning, then insert instances one layer at a time into a composite scene through inversion-based noise blending.
Key results
- RGBA generator achieves KID 0.0150, IoU 0.892 and CLIP score 18.49, the best on all three metrics versus SD-v1.5, PixArt-α, matting variants, Text2Layer and LayerDiffusion.
- Removing the paired training procedure raises KID from 0.0150 to 0.0181 and drops IoU to 0.814, the largest ablation effect.
- Scene composition qualitatively outperforms GLIGEN, MultiDiffusion and Instance Diffusion on attribute assignment and overlapping objects; results are shown visually only, without quantitative scene metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate RGBA instances with the proposed model instead of PixArt-α | RGBA instance generation · RGBA instance generation | KID: 0.0447 -> 0.0150 | Helped |
| Generate RGBA instances with the proposed model instead of PixArt-α + MA | RGBA instance generation · RGBA instance generation | IoU: 0.811 -> 0.892 | Helped |
| Generate RGBA instances with the proposed model instead of SD-v1.5 + MA | RGBA instance generation · RGBA instance generation | CLIP Score: 18.40 -> 18.49 | Helped |
| Add paired training (ablation) | RGBA instance generator · RGBA instance generation | KID: 0.0181 -> 0.0150 | Helped |
| Add paired training (ablation) | RGBA instance generator · RGBA instance generation | IoU: 0.814 -> 0.892 | Helped |
| Add paired training (ablation) | RGBA instance generator · RGBA instance generation | CLIP Score: 18.41 -> 18.49 | Helped |
| Add conditional sampling (ablation) | RGBA instance generator · RGBA instance generation | KID: 0.0152 -> 0.0150 | Helped |
| Add conditional sampling (ablation) | RGBA instance generator · RGBA instance generation | IoU: 0.887 -> 0.892 | Helped |
| Add conditional sampling (ablation) | RGBA instance generator · RGBA instance generation | CLIP Score: 18.49 -> 18.49 | No clear effect |
| Use large KL regularisation (w_KL=1) instead of very small KL regularisation (w_KL=10^{-6}) (ablation) | RGBA VAE · RGBA generation | image quality | Helped |
| Disentangle RGB and alpha channels instead of using a joint latent representation (ablation) | RGBA VAE · RGBA generation | image quality | Helped |
| Increase VAE latent channels from four to eight (ablation) | RGBA VAE · VAE reconstruction | reconstruction capability | Helped |
Surprising findings
- LayerDiffusion obtained a lower IoU (0.320) than matting-based baselines, attributed to instances generated without transparency.
- Adding Matte Anything to PixArt-α worsened KID (0.0447 to 0.0558) even though it improved IoU.
Limitations
- RGBA generator trained on a relatively small dataset without intrinsically transparent training data.
- Full fine-tuning including the VAE makes interaction with other pretrained diffusion models harder; mutual conditioning sampling increases inference cost.
- Independent instance generation makes assembling coherent scenes harder and requires prompt engineering.
Experimental setup
Datasets: MuLAn AIM-500 HIM-2k Interactive Human Matting RWP-636 PPM-100 AM-2k
Benchmarks: MuLAn-style RGBA instance evaluation (KID, IoU vs ICON masks, CLIP score)
Metrics: KID IoU (Jaccard) CLIP Score
Baselines: Stable Diffusion 1.5 PixArt-α Matte Anything Text2Layer LayerDiffusion GLIGEN MultiDiffusion Instance Diffusion
Builds on: PixArt-α MuLAn LayerDiffusion Text2Layer MultiDiffusion
Closest papers by meaning
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.89)
- Composing Parts for Expressive Object Generation (CVPR 2025 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.88)
- Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation (NeurIPS 2024 · similarity 0.88)
- MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation (CVPR 2025 · similarity 0.88)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.