Research map / Detection and segmentation / CNN features
ROICtrl: Boosting Instance Control for Visual Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROICtrl, a diffusion adapter for regional instance control built on a new ROI-Unpool operation (complement to ROI-Align), and introduces ROICtrl-Bench for free-form instance captions.
Problem
Text-based diffusion models struggle to associate positions and attributes with multiple instances; prior instance-control methods either inject coordinates implicitly (attribute leakage, poor alignment) or use attention masks on full high-resolution maps (high compute).
Approach
Crop ROI features from the high-resolution spatial map with ROI-Align, process them with the pretrained cross-attention plus ROI self-attention, then paste them back with ROI-Unpool and blend with the global attention via learnable fusion weights, avoiding masked attention and learnable caption modules.
Key results
- ROICtrl achieves ROICtrl-Bench mIoU 0.652 and accuracy 48.7% vs. Instance Diffusion 0.607 / 45.6%, and MIGC 0.490 / 38.9%.
- On InstDiff-Bench, ROICtrl reaches AP 41.0 vs. 40.1 for Instance Diffusion, and AP_s 16.3 vs. 10.4.
- ROICtrl runs at 13.1 s/img vs. 129.2 s/img for Instance Diffusion (about 9.8x speedup) at 1024x1024 on A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ROICtrl instead of Instance Diffusion | Instance control models · ROICtrl-Bench AVG | mIoU: 0.607 -> 0.652 | Helped |
| Use ROICtrl instead of Instance Diffusion | Instance control models · ROICtrl-Bench AVG | Acc (%): 45.6 -> 48.7 | Helped |
| Use ROICtrl instead of MIGC | Instance control models · MIG-Bench AVG | mIoU: 0.56 -> 0.66 | Helped |
| Use ROICtrl instead of Instance Diffusion | Instance control models · InstDiff-Bench | AP: 40.1 -> 41.0 | Helped |
| Replace ROI-Align and ROI-Unpool with masked attention (ablation) | ROICtrl · ROICtrl-Bench AVG | mIoU: 0.652 -> 0.628 | Hurt |
| Replace ROI-Align and ROI-Unpool with masked attention (ablation) | ROICtrl · ROICtrl inference | Speed (s/img): 13.1 -> 31.5 | Hurt |
| Add ROI self-attention (ablation) | ROICtrl · ROICtrl-Bench AVG | mIoU: 0.540 -> 0.652 | Helped |
| Add ROI self-attention (ablation) | ROICtrl · InstDiff-Bench | AP: 32.7 -> 41.0 | Helped |
| Add Lreg regularization (ablation) | ROICtrl · InstDiff-Bench | Color Acc: 58.2 -> 62.3 | Helped |
| Add Lreg regularization (ablation) | ROICtrl · InstDiff-Bench | Texture Acc: 21.9 -> 29.3 | Helped |
| Replace global coordinates with local coordinates (ablation) | ROICtrl · ROICtrl-Bench AVG | mIoU: 0.652 -> 0.655 | No clear effect |
| Replace global coordinates with local coordinates (ablation) | ROICtrl · InstDiff-Bench | AP: 41.0 -> 42.1 | Helped |
Surprising findings
- Trained only on MS-COCO (118K samples), ROICtrl slightly underperforms Instance Diffusion (trained on 5M internal images) on out-of-distribution instance captions in ROICtrl-Bench.
Limitations
- Trained only on MS-COCO, weaker on out-of-distribution instance captions than Instance Diffusion.
- Bounding boxes give only coarse spatial cues, requiring add-ons for fine-grained control.
Experimental setup
Datasets: MS-COCO ROICtrl-Bench
Benchmarks: ROICtrl-Bench MIG-Bench InstDiff-Bench
Metrics: mIoU accuracy AP AP50 CLIP score inference speed memory
Baselines: GLIGEN MIGC Instance Diffusion ROICtrl (mask) GLIGEN*
Builds on: GLIGEN ROI-Align Stable Diffusion ControlNet IP-Adapter
Closest papers by meaning
- InstanceDiffusion: Instance-level Control for Image Generation (CVPR 2024 · similarity 0.88)
- MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis (CVPR 2024 · similarity 0.87)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.86)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.86)
- OmniGen: Unified Image Generation (CVPR 2025 · similarity 0.86)
- Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.