Research map / Detection and segmentation / Open-set and few-shot detection
DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiverGen, a generative data augmentation pipeline for LVIS instance segmentation that scales synthetic data to millions by diversifying categories, prompts (ChatGPT-generated), and generators (Stable Diffusion + DeepFloyd-IF).
Problem
Instance segmentation on long-tailed data (LVIS) is data-hungry and models overfit rare categories; existing generative data augmentation does not fully exploit generative models and does not scale well to large generated datasets.
Approach
Explains generative data via distribution discrepancy (it widens the learned data distribution and reduces train-val gap) and scales it to millions of images by enhancing diversity along category, prompt, and generator axes, combined with a four-stage pipeline (SAM-background annotation, CLIP inter-similarity filtering, instance paste).
Key results
- DiverGen with Swin-L reaches 51.2 box AP and 45.5 mask AP on LVIS val, +1.1 box AP and +1.1 mask AP over X-Paste (50.1/44.4).
- On LVIS rare categories DiverGen achieves 50.1 box APr and 45.8 mask APr, +1.9 and +2.5 over X-Paste.
- Using 600k generated images, diversity enhancement (GDDE) raises mask AP by 0.64 and box AP by 0.55; at 1.2M images GDDE gives +1.21 box AP over the non-diverse 1.2M setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiverGen instead of X-Paste | Swin-L instance segmentation on LVIS · LVIS val | box AP: +1.1 box AP | Helped |
| Use DiverGen instead of X-Paste | Swin-L instance segmentation on LVIS · LVIS val | mask AP: +1.1 mask AP | Helped |
| Add generative data to real training data (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 47.50 -> 49.65 | Helped |
| Increase manually prompted IF data from 600k to 1200k without GDDE (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 50.03 -> 49.44 | Hurt |
| Add GDDE to the 1200k generative dataset (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 49.44 -> 51.24 | Helped |
| Add 50 extra ImageNet categories (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 49.44 -> 49.92 | Helped |
| Add 250 extra ImageNet categories (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 49.44 -> 50.59 | Helped |
| Increase extra ImageNet categories from 250 to 566 (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 50.59 -> 50.35 | Hurt |
| Increase prompt count from 1 to 32 per category (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 49.65 -> 50.03 | Helped |
| Increase prompt count from 32 to 128 per category (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 50.03 -> 50.27 | Helped |
| Use Stable Diffusion-generated data instead of no generative data (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 47.50 -> 48.13 | Helped |
| Mix Stable Diffusion and DeepFloyd-IF generated data (ablation) | CenterNet2 with Swin-L · LVIS val | box AP: 49.44 -> 50.78 | Helped |
Surprising findings
- Scaling generative data quantity without diversity saturates or degrades performance: 1,200k manually-prompted images underperform 600k.
- Generative data from a single model alone shows that the model trained on one category group performs worst on that group's categories, indicating insufficient diversity misleads learning.
Limitations
- Relies on ChatGPT prompts only for a subset of categories due to inference cost.
- Extra ImageNet categories beyond ~250 degrade performance, so category diversity has an upper limit.
- Evaluated only on LVIS with a single baseline architecture (CenterNet2 with Swin-L).
Experimental setup
Datasets: LVIS ImageNet-1K Stable Diffusion generated images DeepFloyd-IF generated images
Benchmarks: LVIS
Metrics: box AP mask AP rare category box AP rare category mask AP train-val gap KL divergence
Baselines: CenterNet2 X-Paste Detic Copy-Paste Tan et al.
Builds on: X-Paste CenterNet2 SAM CLIP
Closest papers by meaning
- A Generative Approach at the Instance-Level for Image Segmentation Under Limited Training Data Conditions (Student Abstract) (AAAI 2025 · similarity 0.87)
- UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation (NeurIPS 2025 · similarity 0.87)
- Image Generation Diversity Issues and How to Tame Them (CVPR 2025 · similarity 0.86)
- Training-Free Dataset Pruning for Instance Segmentation (ICLR 2025 · similarity 0.86)
- Generative Active Learning for Long-tailed Instance Segmentation (ICML 2024 · similarity 0.86)
- Scaling up Image Segmentation across Data and Tasks (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.