Research map / Detection and segmentation / Open-set and few-shot detection
A Training-free Synthetic Data Selection Method for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free synthetic data selection pipeline (PCS for image quality via patch-mixed CLIP similarity drop, ASF for annotation quality via CLIP-generated reference masks) that keeps about half of Dataset-Diffusion synthetic samples and improves segmenter mIoU on VOC 2012 and COCO.
Problem
Synthetic image-annotation pairs for semantic segmentation contain many low-quality samples (distribution mismatch, wrong annotations), and training on all of them harms segmenter performance.
Approach
Score synthetic images by the CLIP text-to-image similarity drop after patch-mixing (PCS), and filter annotations by class-grouped mIoU against reference masks derived from CLIP softmax CAMs (ASF), with no training required for selection.
Key results
- On VOC 2012 with DeepLabV3 ResNet101, SDS-VOC (26k samples) reaches 59.1% mIoU versus 56.9% for Dataset Diffusion on 40k samples (reproduced).
- With CDL ResNet101 on VOC, SDS reaches 62.5% mIoU vs 59.6% for Dataset Diffusion.
- On MS COCO 2017 with CDL ResNet101, SDS-COCO (50k) reaches 33.4% mIoU vs 30.3% for Dataset Diffusion (80k).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select samples with SDS instead of using all Dataset Diffusion samples | CDL, ResNet101 · PASCAL VOC 2012 | mIoU (%): 59.6* -> 62.5 | Helped |
| Select samples with SDS instead of using all Dataset Diffusion samples | CDL, ResNet101 · MS COCO 2017 | mIoU (%): 30.3* -> 33.4 | Helped |
| Add PCS to the synthetic dataset (ablation) | DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 58.1 -> 59.5 | Helped |
| Add ASF after PCS (ablation) | DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 59.5 -> 60.4 | Helped |
| Use patch scale {8} instead of multiple patch scales (ablation) | PCS with DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 60.4 -> 59.9 | Hurt |
| Use patch scale {16} instead of multiple patch scales (ablation) | PCS with DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 60.4 -> 60.2 | Hurt |
| Use patch scale {32} instead of multiple patch scales (ablation) | PCS with DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 60.4 -> 59.5 | Hurt |
| Group annotations by number of classes rather than applying direct selection (ablation) | ASF with DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 57.5 -> 59.2 | Helped |
| Group annotations by object class rather than applying direct selection (ablation) | ASF with DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 57.5 -> 59.6 | Helped |
| Apply both ASF class-balance rules rather than direct selection (ablation) | ASF with DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 57.5 -> 60.4 | Helped |
| Use 30k selected samples instead of 40k (ablation) | SDS with DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 58.1 -> 59.7 | Helped |
| Use 26k selected samples instead of 40k (ablation) | SDS with DeepLabV3, ResNet50 · PASCAL VOC 2012 | mIoU (%): 58.1 -> 60.4 | Helped |
Surprising findings
- Synthetic images generally receive high text-to-image CLIP similarity, so raw similarity alone is not a usable quality signal.
- Using only 15k of the synthetic samples drops performance to 58.0%, below the 26k selected subset (60.4%), showing that selection quality matters more than size alone.
Limitations
- Relies on hyperparameter thresholds (tau_s=0.8, tau_PCS=0.1) and top-60% retention chosen empirically.
- Evaluated only with Dataset Diffusion-generated data and a few segmenters; generalization to other generators is not shown.
- Reference annotations come from CLIP-derived CAMs, which may themselves be noisy.
Experimental setup
Datasets: PASCAL VOC 2012 SBD MS COCO 2017 Dataset Diffusion synthetic data
Benchmarks: PASCAL VOC 2012 MS COCO 2017
Metrics: mIoU
Baselines: Dataset Diffusion DiffuMask VOC's training COCO's training
Builds on: Dataset-Diffusion CLIP DeepLabV3 Mask2Former DiffuMask
Closest papers by meaning
- Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.85)
- Scaling Laws of Synthetic Images for Model Training ... for Now (CVPR 2024 · similarity 0.85)
- A Generative Approach at the Instance-Level for Image Segmentation Under Limited Training Data Conditions (Student Abstract) (AAAI 2025 · similarity 0.84)
- ImagineFSL: Self-Supervised Pretraining Matters on Imagined Base Set for VLM-based Few-shot Learning (CVPR 2025 · similarity 0.84)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.84)
- Provably Improving Generalization of Few-shot models with Synthetic Data (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.