Research map / Image generation / Autoregressive and latent generation
MaskFactory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskFactory, a two-stage pipeline that edits DIS ground-truth masks (rigid viewpoint edits via Zero123, non-rigid edits via MasaCtrl-style attention with topology-preserving adversarial training) and then generates matching RGB images with a ControlNet-style multi-condition (mask + Canny + prompt) diffusion model, to create synthetic data for dichotomous image segmentation.
Problem
Creating precise image-mask training data for dichotomous image segmentation (DIS) is labor-intensive and costly, and existing diffusion-based synthetic data methods suffer from scene deviation, attention-map noise causing mask errors, and limited sample diversity.
Approach
Edit existing ground-truth masks with combined rigid (zero-shot viewpoint change via Zero123) and non-rigid (attention-based editing with a topology-preserving adversarial discriminator on edge graphs) operations, then synthesize paired RGB images conditioned on the edited mask and its Canny edges, instead of relying on attention-map pseudo-labels.
Key results
- On DIS-VD, IS-Net trained with 10,000 MaskFactory samples reaches max F1 0.835 vs 0.791 without, and DatasetDM/Dataset Diffusion degrade beyond 5,000 generated images.
- On DIS-TE1, max F1 improves for IS-Net (0.740 to 0.784), FP-DIS (0.784 to 0.805), UDUN (0.784 to 0.799), BiRefNet (0.866 to 0.882) and SAM-HQ (0.897 to 0.905) when adding generated data.
- Mixed rigid+non-rigid editing gives max F1 0.784 vs 0.768 (rigid) and 0.771 (non-rigid) in ablation; CLIP similarity to real data is 0.8967 for All vs 0.6683 for DatasetDM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train IS-Net with 10000 MaskFactory-generated image pairs | IS-Net · DIS-VD | max F1: 0.791 -> 0.835 | Helped |
| Train IS-Net with 10000 MaskFactory-generated image pairs | IS-Net · DIS-TE1 | max F1: 0.740 -> 0.784 | Helped |
| Train IS-Net with 10000 MaskFactory-generated image pairs | IS-Net · DIS-TE2 | max F1: 0.799 -> 0.822 | Helped |
| Train IS-Net with 10000 MaskFactory-generated image pairs | IS-Net · DIS-TE3 | max F1: 0.830 -> 0.870 | Helped |
| Train IS-Net with 10000 MaskFactory-generated image pairs | IS-Net · DIS-TE4 | max F1: 0.827 -> 0.879 | Helped |
| Add mixed rigid and non-rigid mask editing instead of rigid-only editing (ablation) | MaskFactory · DIS-VD | max F1: 0.768 -> 0.784 | Helped |
| Add mixed rigid and non-rigid mask editing instead of non-rigid-only editing (ablation) | MaskFactory · DIS-VD | max F1: 0.771 -> 0.784 | Helped |
| Add content and structure losses to GAN loss (ablation) | MaskFactory mask editing · DIS-VD | max F1: 0.778 -> 0.784 | Helped |
| Add GAN and structure losses to content loss (ablation) | MaskFactory mask editing · DIS-VD | max F1: 0.745 -> 0.784 | Helped |
| Add GAN and content losses to structure loss (ablation) | MaskFactory mask editing · DIS-VD | max F1: 0.751 -> 0.784 | Helped |
| Add prompt and Canny conditions to the mask condition (ablation) | MaskFactory image generation · DIS-VD | max F1: 0.778 -> 0.784 | Helped |
| Add Canny condition to mask and prompt conditions (ablation) | MaskFactory image generation · DIS-VD | max F1: 0.782 -> 0.784 | Helped |
Surprising findings
- Non-rigid-only editing achieves the highest CLIP and UMAP similarity to real data (0.9147) even though mixed editing gives the best segmentation F1.
Limitations
- ControlNet-based generation sometimes yields unnatural images with stark foreground-background contrast needing harmonization.
- Relies on pre-annotated image-mask pairs, so it cannot generate data autonomously.
- Complex scenes can produce unrealistic element placement.
Experimental setup
Datasets: DIS5K (DIS-TR, DIS-VD, DIS-TE1-4)
Benchmarks: DIS5K DIS-VD DIS5K DIS-TE1 DIS5K DIS-TE2 DIS5K DIS-TE3 DIS5K DIS-TE4
Metrics: max F1 weighted F-measure MAE S-measure E-measure CLIP similarity UMAP distribution
Baselines: DatasetDM Dataset Diffusion IS-Net FP-DIS UDUN BiRefNet SAM-HQ
Builds on: Zero123 MasaCtrl ControlNet DatasetDM Dataset Diffusion
Closest papers by meaning
- High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity (ICLR 2025 · similarity 0.86)
- Click2Mask: Local Editing with Dynamic Mask Generation (AAAI 2025 · similarity 0.86)
- UniGS: Unified Representation for Image Generation and Segmentation (CVPR 2024 · similarity 0.86)
- $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.86)
- MaskDiff: Modeling Mask Distribution with Diffusion Probabilistic Model for Few-Shot Instance Segmentation (AAAI 2024 · similarity 0.86)
- Multi-view Aggregation Network for Dichotomous Image Segmentation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.