Research map / Label-efficient and robust learning / Recognition and long-tail
Stratify or Die: Rethinking Data Splits in Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPS (iterative pixel-level stratification) and WDES (Wasserstein-driven genetic algorithm) to create label-representative train/test and K-fold splits for image segmentation, and shows WDES yields more representative folds and lower cross-validation variance than random splitting.
Problem
Random splitting of segmentation datasets yields folds with shifted pixel-level class distributions, producing biased and high-variance evaluations, especially for small, imbalanced datasets; existing multi-label stratification does not account for pixel-level class proportions.
Approach
Stratify image-level splits using pixel-count label distributions: IPS adapts iterative stratification to pixel quotas, and WDES evolves fold assignments with a genetic algorithm minimizing a Wasserstein-based label distance (LWD), with a claimed convergence bound.
Key results
- WDES achieves PLD and LWD lower than random splitting and IPS on all five datasets (e.g., PascalVOC PLD 456 vs 955 random; Cityscapes LWD 217 vs 304 random).
- In 10-fold UNet cross-validation, WDES has the best average rank in standard deviation of accuracy, F1, and IoU (rank 1.6 vs 2.4/1.8 for random).
- The advantage of WDES over random splitting shrinks on high-entropy datasets (LoveDA, Cityscapes) where random splitting has lower variance than WDES.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WDES instead of random splitting | Segmentation dataset splits · PascalVOC | PLD: 955 -> 456 | Helped |
| Use WDES instead of random splitting | Segmentation dataset splits · CamVid | PLD: 63.4 -> 36.7 | Helped |
| Use WDES instead of random splitting | Segmentation dataset splits · EndoVis | PLD: 609 -> 208 | Helped |
| Use WDES instead of random splitting | Segmentation dataset splits · LoveDA | PLD: 1110 -> 377 | Helped |
| Use WDES instead of random splitting | Segmentation dataset splits · Cityscapes | PLD: 126 -> 67 | Helped |
| Use WDES instead of random splitting | Segmentation dataset splits · Five benchmark datasets | LWD | Helped |
| Use WDES instead of random splitting | UNet 10-fold evaluation · Low-entropy datasets | Performance variance (accuracy, F1, and IoU) | Helped |
| Use random splitting instead of WDES | UNet 10-fold evaluation · LoveDA | Accuracy standard deviation: 7.49 -> 6.63 | Helped |
| Use random splitting instead of WDES | UNet 10-fold evaluation · High-entropy datasets | Performance variance | Mixed |
| Use IPS instead of WDES | Segmentation dataset splits · Five benchmark datasets | PLD and LWD | Hurt |
| Use IPS instead of random splitting | UNet 10-fold evaluation · Five benchmark datasets | Performance variance | Mixed |
Surprising findings
- IPS, despite being designed for pixel-level proportions, performs worst on split-quality and variance ranks, below random splitting.
- On higher-entropy datasets such as LoveDA, random splitting shows lower performance variance than WDES.
Limitations
- Relies on Wasserstein distance of label distributions, ignoring spatial structure and class co-occurrence.
- Convergence theorem guarantees optimality only within the explored subset; the global optimality claim in the abstract is weaker than stated.
- Variance reduction is not shown to come from test-set consistency versus training; empirical validation of unbiased evaluation left to future work.
Experimental setup
Datasets: Cityscapes CamVid EndoVis2018 LoveDA PascalVOC 2012
Benchmarks: Cityscapes CamVid EndoVis2018 LoveDA PascalVOC 2012
Metrics: SD PLD LWD accuracy F1 IoU standard deviation across folds
Baselines: Random splitting (KFold) IPS Iterative Stratification
Builds on: Iterative Stratification Evosplit scikit-learn KFold UNet
Closest papers by meaning
- Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation (NeurIPS 2026 · similarity 0.88)
- Improving Semi-Supervised Semantic Segmentation with Sliced-Wasserstein Feature Alignment and Uniformity (CVPR 2025 · similarity 0.85)
- $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.84)
- Scaling up Image Segmentation across Data and Tasks (CVPR 2025 · similarity 0.84)
- Learning Hierarchical Image Segmentation For Recognition and By Recognition (ICLR 2024 · similarity 0.84)
- Annotation Ambiguity Aware Semi-Supervised Medical Image Segmentation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.