Research map / Image generation / Text-to-image diffusion
InstanceDiffusion: Instance-level Control for Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds instance-level control (per-instance text plus points, scribbles, boxes, or masks) to a frozen text-to-image diffusion UNet via UniFusion, ScaleU, and a Multi-instance Sampler, trained on automatically annotated instance captions.
Problem
Text-to-image diffusion models lack precise control over individual instances' locations and attributes; existing location-conditioned methods support only one location format and lack per-instance captions.
Approach
Unifies heterogeneous location formats (points, scribbles, boxes, masks) as sets of 2D points embedded with Fourier features and fused with per-instance CLIP text tokens via instance-masked self-attention and gated addition into a frozen UNet; adds ScaleU learnable channel-wise rescaling of backbone and skip features and a Multi-instance Sampler that denoises each instance separately before global
Key results
- On COCO val with box inputs, InstanceDiffusion reaches 38.8 AP box and 55.4 AP50 box, +19.2 AP and +20.4 AP50 over GLIGEN.
- For mask inputs, IoU of 61.6 (vs. 35.0 for DenseDiffusion) and AP mask 50 of 50.0 (vs. 13.8 for ControlNet).
- Color attribute accuracy rises to 54.4 vs. 19.2 for GLIGEN; humans prefer InstanceDiffusion outputs 80.3% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add instance-level captions during training (ablation) | InstanceDiffusion · COCO val instance attribute binding | Color accuracy: 38.2(17.2) -> 55.4 | Helped |
| Add ScaleU (ablation) | InstanceDiffusion · COCO val, mask inputs | APmask50: 47.7(2.3) -> 50.0 | Helped |
| Use Format Aware fusion instead of Joint format fusion (ablation) | InstanceDiffusion · COCO val, mask inputs | APmask50: 45.5(5.5) -> 50.0 | Helped |
| Add Instance-Masked Attention (ablation) | InstanceDiffusion · COCO val, instance attribute binding | Color accuracy: 53.1(2.3) -> 55.4 | Helped |
| Add Multi-instance Sampler (ablation) | InstanceDiffusion · COCO val, instance attribute binding | Color accuracy: 49.5(5.9) -> 55.4 | Helped |
| Add Multi-instance Sampler (ablation) | InstanceDiffusion · COCO val, mask inputs | FID: 28.6(3.1) -> 25.5 | Helped |
| Add extra tokens from binary instance masks (ablation) | InstanceDiffusion · COCO val, mask inputs | APmask50: 46.7 -> 50.0 | Helped |
| Parameterize masks with boundary and inside points (ablation) | InstanceDiffusion · COCO val, mask inputs | APmask50: 47.5 -> 50.0 | Helped |
| Increase mask points from 64 to 128 (ablation) | InstanceDiffusion · COCO val, mask inputs | APmask50: 45.7 -> 48.5 | Helped |
| Use 512 instead of 256 points per mask (ablation) | InstanceDiffusion · COCO val, mask inputs | APmask50: 50.0 -> 50.0 | No clear effect |
| Replace ScaleU with FreeU (ablation) | InstanceDiffusion · COCO val, box inputs | APbox50: 52.2 -> 55.4 | Hurt |
| Add box and mask formats alongside point inputs (ablation) | InstanceDiffusion · COCO val, multiple location formats | APbox50: - -> 38.8 -> 44.6 | Helped |
Surprising findings
- Texture binding remains hard for all methods, including InstanceDiffusion.
- Feeding masks as 64x64 semantic channels (prior practice) hurt performance on overlapping and small instances, so per-instance mask tokens were used instead.
Limitations
- Performance on small objects lags behind larger objects.
- Texture binding is challenging for all tested methods.
- Multi-instance Sampler adds inference cost per instance.
Experimental setup
Datasets: internal licensed natural image dataset (5M images) with auto-generated instance annotations COCO val LVIS val COCO 250-sample subset
Benchmarks: COCO val LVIS val COCO 250-sample subset
Metrics: AP box AP50 box AR box AP mask IoU FID Points in Mask (PiM) color accuracy texture accuracy local CLIP-score
Baselines: GLIGEN ControlNet DenseDiffusion SpaText
Builds on: GLIGEN FreeU Stable Diffusion RAM Grounded-SAM
Closest papers by meaning
- MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis (CVPR 2024 · similarity 0.89)
- ROICtrl: Boosting Instance Control for Visual Generation (CVPR 2025 · similarity 0.88)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.88)
- Local Conditional Controlling for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.88)
- One Diffusion to Generate Them All (CVPR 2025 · similarity 0.87)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.