Research map / Image generation / Text-to-image diffusion
MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIGC, which decomposes multi-instance text-to-image generation into per-instance shading subtasks inside Stable Diffusion's cross-attention layers, and introduces the COCO-MIG benchmark.
Problem
Text-to-image diffusion models cannot reliably generate multiple instances in one image with correct per-instance positions, attributes, and counts (textual and spatial attribute leakage).
Approach
Divide-and-conquer: treat each instance as a shading subtask in cross-attention space, enhance each with a trainable enhancement attention layer using grounded phrase tokens (text + Fourier box embedding), add a layout attention template and global background, and fuse them with a Shading Aggregation Controller.
Key results
- On COCO-MIG, instance success rate rises from 32.39% (GLIGEN) to 58.43% and mIoU from 32.25 to 51.48.
- On COCO-Position, success ratio rises from 70.52% to 80.29% and AP from 40.68 to 54.69 over GLIGEN, with FID comparable to Stable Diffusion (24.52 vs 23.56).
- On DrawBench, attribute success rate increases from 48.20% to 97.50%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MIGC instead of GLIGEN | text-to-image generation · COCO-MIG | Instance Success Rate (Avg, %): 32.39 -> 58.43 | Helped |
| use MIGC instead of GLIGEN | text-to-image generation · COCO-Position | AP: 40.68/68.26/42.85 to 54.69/84.17/61.71 | Helped |
| use MIGC instead of GLIGEN | text-to-image generation · DrawBench | Attribute (mechanical, %): 51.00 -> 79.00 | Helped |
| use MIGC instead of GLIGEN | text-to-image generation · DrawBench | Spatial (mechanical, %): 61.25 -> 69.38 | Helped |
| add Shading Aggregation Controller (SAC) (ablation) | MIGC · COCO-Position | AP: 52.05 -> 54.69 | Helped |
| add Enhancement Attention (EA) (ablation) | MIGC · COCO-Position | Success Ratio: 12.10 -> 80.16 | Helped |
| add Layout Attention (LA) (ablation) | MIGC with SAC · COCO-Position | AP: 53.03 -> 54.69 | Helped |
| add inhibition loss with weight 0.1 (ablation) | MIGC · COCO-Position | AP: 52.46 -> 54.69 | Helped |
| set inhibition loss weight to 1.0 instead of 0.1 (ablation) | MIGC · COCO-Position | AP: 54.69 -> 55.62 | Helped |
| set inhibition loss weight to 1.0 instead of 0.1 (ablation) | MIGC · COCO-Position | FID: 24.52 -> 26.94 | Hurt |
| remove inhibition loss (ablation) | MIGC · COCO-Position | AP: 54.69 -> 52.46 | Hurt |
Surprising findings
- Inhibition loss weight 1.0 improves AP further but slightly degrades FID, so weight 0.1 was chosen.
Limitations
- Interaction between instances is not controlled (stated as future work)
- Relies on automatic detection (Grounding-DINO, stanza) to build training instance annotations
Experimental setup
Datasets: COCO 2014 COCO-Position DrawBench
Benchmarks: COCO-MIG COCO-Position DrawBench
Metrics: Instance Success Rate mIoU AP AP50 AP75 Success Ratio CLIP score Local CLIP score FID-6K Attribute success rate
Baselines: Stable Diffusion v1.4 GLIGEN Multi-Diffusion Layout Diffusion Box-Diffusion TFLCG AAE Structure Diffusion
Builds on: Stable Diffusion GLIGEN Multi-Diffusion Box-Diffusion
Closest papers by meaning
- InstanceDiffusion: Instance-level Control for Image Generation (CVPR 2024 · similarity 0.89)
- ROICtrl: Boosting Instance Control for Visual Generation (CVPR 2025 · similarity 0.87)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.86)
- Generating compositional scenes via Text-to-image RGBA Instance Generation (NeurIPS 2024 · similarity 0.86)
- Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis (NeurIPS 2024 · similarity 0.84)
- MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.