Research map / Detection and segmentation / Open-set and few-shot detection
FlexDataset: Crafting Annotated Dataset Generation for Diverse Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlexDataset, a composition-to-data framework that trains a layout-conditioned diffusion model (MIMC C2I) on a small labeled subset and uses lightweight tuned perception decoders on its latent queries to produce pixel-level annotations for synthetic datasets.
Problem
Generating pixel-level annotated synthetic datasets for perception tasks is limited by text-based generation, which struggles with complex multi-instance, multi-category scenes with precise spatial layouts, and by slow annotation via full perception models.
Approach
Replace text prompts with user-specified bounding-box compositions and categories fed into a linear-attention MIMC encoder with instance-masked attention, and generate annotations by passing the diffusion latent query directly into tuned perception decoders (VAG Plan A) rather than running full perception encoders on generated images.
Key results
- On VOC 2012 semantic segmentation with Swin-B, 100 real + 40k synthetic images, FlexDataset reaches 81.6 mIoU vs DatasetDM 78.5.
- On COCO2017 instance segmentation with Swin-B and 800 real images, FlexDataset reaches 31.4 AP vs DatasetDM 26.5.
- VAG Plan A (tuned perception decoders) cuts semantic segmentation synthesis time from 257.3h to 138.4h with comparable mIoU (81.1 vs 81.6), and depth estimation synthesis from 92.3h to 18.6h.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FlexDataset synthetic data instead of DatasetDM synthetic data | Swin-B semantic segmentation · VOC 2012 | mIoU: 78.5 -> 81.6 | Helped |
| Use FlexDataset synthetic data instead of DatasetDM synthetic data | Swin-B instance segmentation · COCO val2017 | AP: 26.5 -> 31.4 | Helped |
| Use FlexDataset synthetic data instead of DatasetDM synthetic data | Depth estimation · NYU Depth V2 | REL: 0.21 -> 0.17 | Helped |
| Use FlexDataset synthetic data instead of real-only training data | MODNet · PPM-100 | MSE: 0.44 -> 0.24 | Helped |
| Use FlexDataset synthetic data instead of DatasetDM synthetic data | Zero-shot semantic segmentation · VOC 2012 | mIoU: 68.4 -> 80.4 | Helped |
| Use FlexDataset synthetic data instead of DatasetDM synthetic data | Long-tail semantic segmentation · VOC 2012 | mIoU: 70.0 -> 72.7 | Helped |
| Add tuned perception decoders instead of using entire perception models for annotation synthesis (ablation) | FlexDataset annotation generation · VOC 2012 semantic segmentation | mIoU: 81.1 -> 81.6 | Helped |
| Add tuned perception decoders instead of using entire perception models for annotation synthesis (ablation) | FlexDataset annotation generation · VOC 2012 semantic segmentation annotation synthesis | synthesis time: 257.3 -> 138.4 | Helped |
| Add tuned perception decoders instead of using entire perception models for annotation synthesis (ablation) | FlexDataset annotation generation · NYU Depth V2 depth estimation | depth estimation score: 0.20 -> 0.17 | Helped |
| Add tuned perception decoders instead of using entire perception models for annotation synthesis (ablation) | FlexDataset annotation generation · NYU Depth V2 depth estimation annotation synthesis | synthesis time: 92.3 -> 18.6 | Helped |
| Vary the proportion of single-category and multi-category instances in synthetic images (ablation) | FlexDataset · VOC 2012 semantic segmentation | mIoU | No clear effect |
| Add slight bounding-box offsets to synthetic image compositions (ablation) | FlexDataset · VOC 2012 semantic segmentation | mIoU | Helped |
Limitations
- Relies on comparisons largely against DatasetDM; many baselines results are taken from prior work rather than re-run.
- Text prompt supervision gave only a small (0.6%) gain, and the approach depends on a small labeled subset for reverse tuning.
Experimental setup
Datasets: COCO 2017 Stuff Segmentation PASCAL VOC 2012 NYU Depth V2 PPM-100
Benchmarks: PASCAL VOC 2012 COCO2017 NYU Depth V2 PPM-100
Metrics: mIoU AP REL MSE MAD
Baselines: DatasetDM DiffuMask DatasetDiffusion MODNet Mask2Former DetDiffusion Li et al.
Builds on: LayoutDiffusion GLIGEN DatasetDM DiffuMask Depth-Anything
Closest papers by meaning
- FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction (NeurIPS 2025 · similarity 0.85)
- Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation (ICML 2025 · similarity 0.85)
- Low-Biased General Annotated Dataset Generation (CVPR 2025 · similarity 0.84)
- Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.84)
- FlexPose: Pose Distribution Adaptation with Limited Guidance (AAAI 2025 · similarity 0.84)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.