Research map / Detection and segmentation / Open-set and few-shot detection
Generative Active Learning for Long-tailed Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BSGAL, an online, gradient-cache-based method that accepts or rejects batches of Stable Diffusion-generated instances for copy-paste training of long-tailed instance segmentation on LVIS.
Problem
Generated data for long-tailed instance segmentation is of uneven quality and not all of it helps downstream models, yet there is no principled way to select and use generated data, especially under nearly unlimited generated pools.
Approach
Estimate each generated batch's contribution with a first-order gradient dot product between generated-instance loss gradients and a momentum-updated gradient cache of a sampled real test set, and accept or reject the batch online during training.
Key results
- On LVIS with ResNet-50 CenterNet2, BSGAL reaches 35.40 AP_box and 27.95 AP_r_box versus 34.20 and 24.33 for the X-Paste baseline (+1.2 AP_box, +3.62 AP_r_box).
- With Swin-L backbone, BSGAL reaches 50.47 AP_box and 47.55 AP_r_box versus 49.57 and 44.87 for the baseline.
- On CIFAR-10 toy experiment, adding contribution-selected generated data raised accuracy from 86.71 (all generated) to 87.61 over 86.28 with only real data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| gradient-cache contribution-based batch acceptance (BSGAL) vs copy-paste baseline | CenterNet2 R50 on LVIS · LVIS | AP_box: +1.2 | Helped |
| gradient-cache contribution-based batch acceptance (BSGAL) vs copy-paste baseline | CenterNet2 R50 on LVIS · LVIS | AP_r_box: +3.62 | Helped |
| CLIP filtering of generated data | copy-paste baseline with CenterNet2 R50 · LVIS | AP_box: +0.15 | No clear effect |
| random batch-level dropout at same acceptance rate (ablation) | BSGAL R50 on LVIS · LVIS | AP_r_box: -2.90 vs BSGAL | Hurt |
| test set sampled from pasted classes vs all classes (ablation) | BSGAL R50 on LVIS · LVIS | AP_r_box: +1.20 | Helped |
| test set sampled from pasted classes vs all images (ablation) | BSGAL R50 on LVIS · LVIS | AP_r_box: +1.36 | Helped |
| global average pooling for gradient cache (instead of momentum) (ablation) | BSGAL R50 on LVIS · LVIS | AP_r_box: -1.11 AP_r_box, -0.33 AP_box | Hurt |
| test loss restricted to first-stage classification loss (ablation) | contribution estimation in BSGAL · LVIS | AP_box: -0.30 vs all-stage cls | Hurt |
| using all training losses as test loss instead of classification loss (ablation) | contribution estimation in BSGAL · LVIS | AP_box: -0.26 AP_box, -1.23 AP_r_box vs cls | Hurt |
| gradient normalization (cosine similarity) in contribution estimate (ablation) | BSGAL R50 on LVIS · LVIS | AP_box: +0.35 | Helped |
| momentum coefficient beta set to 0.8 (vs 0.1) (ablation) | grad cache in BSGAL · LVIS | AP_box: -0.90 | Hurt |
| offline contribution estimation (fixed model) instead of online (ablation) | copy-paste pipeline on LVIS · LVIS | AP_box | No clear effect |
Surprising findings
- Random batch-level dropout at the same acceptance rate also gives a slight gain, and using the entire training set as the test set (global gradient cache) hurts performance, attributed to reduced data diversity.
- CLIP filtering of generated data yields only 0.1-0.2 AP_box improvement over the unfiltered baseline.
Limitations
- Generated data pipeline depends on a fixed generative model; the generative model is not updated in the loop.
- Computing gradient cache increases GPU memory (9836M vs 6534M baseline per GPU) and the method requires a test-set sampling design choice.
- Evaluated only on LVIS with CenterNet2 and two backbones; authors note generality to other tasks and architectures is untested.
Experimental setup
Datasets: LVIS CIFAR-10 Stable Diffusion V1.5 generated images DeepFloyd-IF generated images Perfusion generated images
Benchmarks: LVIS
Metrics: AP box AP mask AP_r AP_c AP_f
Baselines: X-Paste copy-paste baseline X-Paste + CLIP filtering Loss estimate Grad estimate Random batch-level dropout
Builds on: X-Paste TracIn VeSSAL CenterNet2
Closest papers by meaning
- A Generative Approach at the Instance-Level for Image Segmentation Under Limited Training Data Conditions (Student Abstract) (AAAI 2025 · similarity 0.86)
- DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data (CVPR 2024 · similarity 0.86)
- Density Matters: Improved Core-Set for Active Domain Adaptive Segmentation (AAAI 2024 · similarity 0.84)
- Diffusion-Driven Two-Stage Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2025 · similarity 0.84)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.84)
- FreeGen: Bridging Visual-Linguistic Discrepancies Towards Diffusion-based Pixel-level Data Synthesis (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.