Research map / Image generation / Fast diffusion sampling

$\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D³S², a two-stage diffusion-guided dataset distillation framework for semantic segmentation that selects class-balanced masks greedily and synthesizes aligned images with a layout-to-image diffusion model under segmentation-consistency and class-wise feature guidance.

Problem

Dataset distillation has focused on image classification; extending it to semantic segmentation faces long-tailed class imbalance, the need for pixel-wise image-mask alignment, and high computational cost of per-sample optimization.

Approach

Select a class-balanced mask subset via a greedy submodular-style score favoring underrepresented classes, then generate images conditioned on those masks with a pretrained layout-to-image diffusion model (Freestyle), guided during DDIM sampling by a segmentation-consistency loss and a per-class mean feature matching loss, followed by relabeling with the segmentation model.

Key results

• At 1% on ADE20K, D³S² reaches 24.99% mIoU with Mask2Former (Swin-S), versus 15.65% for Random and 21.42% for Uniform.
• At 1% on COCO-Stuff, D³S² reaches 35.49% mIoU with Swin-S, outperforming Random (29.79%) by 5.70 points.
• At 0.25% on COCO-Stuff with Swin-S, D³S² achieves 26.87% mIoU versus 20.63% for Uniform, against a full-data 48.19%.

Surprising findings

• Matching only first-order (mean) class-wise statistics outperforms mean+variance matching on all six architectures tested.
• Sampling-based guidance on synthesized data gives a gain that is small relative to the choice of mask selection; Stage-I real-image selection alone (18.29%) already beats Uniform.

Limitations

• Performance remains well below full-data training (e.g., 30.28% vs 51.22% at 2% on ADE20K with Swin-S).
• Fine-grained structures such as human hands show generation artifacts from the pretrained layout-to-image model.
• Synthesis requires about 28GB peak memory and roughly 50 seconds per image due to backpropagation through the diffusion model.

Experimental setup

Datasets: ADE20K COCO-Stuff

Benchmarks: ADE20K COCO-Stuff

Metrics: mIoU

Baselines: Random K-Center Herding Uniform FreeMask

Builds on: SRe2L Freestyle DDIM classifier-free guidance IGD FreeMask

Closest papers by meaning

• Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
• High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity (ICLR 2025 · similarity 0.89)
• EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models (ICLR 2024 · similarity 0.89)
• Towards Universal Dataset Distillation via Task-Driven Diffusion (CVPR 2025 · similarity 0.88)
• Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.88)
• Diffusion-Driven Two-Stage Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2025 · similarity 0.88)