Research map / Detection and segmentation / Open-set and few-shot detection
InstaGen: Enhancing Object Detection by Training on Synthetic Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion v1.4 on detection data and adds a GroundingDINO-style grounding head with self-training on novel categories, so the diffusion model synthesizes labeled images used to train Faster R-CNN detectors.
Problem
Object detectors are limited by costly bounding-box annotation and fixed category sets; text-to-image diffusion models produce photo-realistic images but no instance-level boxes and are often simple, few-object scenes.
Approach
Turn a diffusion model into a dataset synthesizer by adding an instance-level grounding head that aligns category text embeddings with the diffusion model's regional features, supervised by an off-the-shelf detector on base classes and extended to novel classes via mean-teacher self-training on features (with dropout instead of image augmentation).
Key results
- On open-vocabulary COCO, InstaGen-trained Faster R-CNN reaches 42.3 AP50 on novel categories versus 37.8 for the best CLIP-based baseline (EdaDet).
- In data-sparse COCO with 10% real data, adding synthetic images raises AP from 23.3 to 28.5.
- Cross-dataset AP50 is 11.4 on Object365 and 23.3 on LVIS, above VL-PLM (10.9 and 22.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add InstaGen synthetic training data | Faster R-CNN, ResNet-50 · COCO open-vocabulary (novel categories) | AP50box: 37.8 -> 42.3 | Helped |
| Add synthetic training images | Faster R-CNN, ResNet-50 · COCO data-sparse (10% real data) | AP: 23.3 -> 28.5 | Helped |
| Add synthetic training images | Faster R-CNN, ResNet-50 · COCO data-sparse (25% real data) | AP: 29.5 -> 32.6 | Helped |
| Add synthetic training images | Faster R-CNN, ResNet-50 · COCO data-sparse (50% real data) | AP: 34.1 -> 35.8 | Helped |
| Add synthetic training images | Faster R-CNN, ResNet-50 · COCO data-sparse (75% real data) | AP: 36.1 -> 37.3 | Helped |
| Add synthetic training images | Faster R-CNN, ResNet-50 · COCO data-sparse (100% real data) | AP: 37.5 -> 38.5 | Helped |
| Use InstaGen-generated training data | COCO-trained object detector · Object365 (cross-dataset) | AP50: 10.9 -> 11.4 | Helped |
| Use InstaGen-generated training data | COCO-trained object detector · LVIS (cross-dataset) | AP50: 22.2 -> 23.3 | Helped |
| Add self-training on novel categories (ablation) | InstaGen grounding head · COCO open-vocabulary (novel categories) | AP50box: 37.1 -> 40.3 | Helped |
| Fine-tune Stable Diffusion on the detection dataset (ablation) | InstaGen · COCO open-vocabulary (novel categories) | AP50box: 40.3 -> 42.3 | Helped |
| Increase generated images per category from 1000 to 3000 (ablation) | InstaGen detector training · COCO open-vocabulary (novel categories) | AP50box: 39.7 -> 42.3 | Helped |
| Set base-category box threshold α to 0.8 rather than 0.7 (ablation) | InstaGen · COCO open-vocabulary (all categories) | AP50box: 51.3 -> 52.3 | Helped |
Surprising findings
- Fine-tuning the diffusion model on detection-data crops improves detector AP by 2.0 over using the off-the-shelf Stable Diffusion images.
- Using a low novel-category score threshold (0.3) drops novel AP to 26.9, showing sensitivity to pseudo-label noise.
Limitations
- Synthetic images focus on clean isolated instances, limiting exposure to real-world clutter and occlusion.
- Diffusion models suffer from long-tail issues and struggle to generate rare categories.
- Relies on an off-the-shelf detector (Mask R-CNN) trained on base categories for pseudo-supervision.
Experimental setup
Datasets: MS-COCO Objects365 LVIS
Benchmarks: open-vocabulary COCO COCO data-sparse subsets Objects365 cross-dataset LVIS cross-dataset
Metrics: AP50 AP
Baselines: Detic PromptDet BARON OADP ViLD F-VLM RO-ViT VLDet CxORA DK-DETR EdaDet VL-PLM
Builds on: Stable Diffusion GroundingDINO DINO Faster R-CNN Mask R-CNN
Closest papers by meaning
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.88)
- SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection (CVPR 2024 · similarity 0.87)
- ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models (NeurIPS 2024 · similarity 0.87)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.87)
- Generalized Diffusion Detector: Mining Robust Features from Diffusion Models for Domain-Generalized Detection (CVPR 2025 · similarity 0.86)
- Revisiting Generative Replay for Class Incremental Object Detection (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.