Research map / Image generation / Fast diffusion sampling
MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGD$^3$, a training-free dataset distillation method that steers a pre-trained class-conditional diffusion model (DiT-XL/2) toward K-means-discovered class modes during early denoising, then stops guidance to preserve quality.
Problem
Generative dataset distillation with diffusion models yields samples concentrated in dense modes, lacking diversity; existing fixes require fine-tuning the diffusion model with distillation losses, which is costly.
Approach
Discover class modes in VAE latent space via K-means, then guide each sample's reverse diffusion toward a distinct mode using a mode-guidance term on the predicted x0, and switch guidance off after t_stop steps (stop guidance) to avoid artifacts, with no model fine-tuning.
Key results
- On ImageNette with DiT, MGD^3 reaches 66.4% at IPC10 (vs 59.1% DiT baseline and 62.0% MinMaxDiff), a reported gain of 4.4 points.
- On ImageNet-100 at IPC20 with ResNetAP-10, MGD^3 reports 33.9% vs 32.3% for MinMaxDiff, while IDC-1 remains slightly better at IPC10 with ResNet-18.
- Mode guidance is reported to take 0.42 hours to generate an ImageNet-100 IPC10 dataset, versus 10 hours for MinMax and over 100 hours for IDC-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mode guidance to DiT | DiT diffusion distillation · ImageNette, IPC 10 | accuracy: 59.1 -> 66.4 | Helped |
| add mode guidance to DiT | DiT diffusion distillation · ImageIDC, IPC 10 | accuracy: 54.1 -> 55.9 | Helped |
| add mode guidance to DiT | DiT diffusion distillation · ImageNet-100, IPC 20 | accuracy: MGD³ exceeds previous SOTA by 1.3%, 1.6%, and 1.4% across the three architectures | Helped |
| add stop guidance after mode guidance (ablation) | MGD³ · ImageNette, IPC 10, ConvNet-6 | accuracy: 57.5 -> 59.6 | Helped |
| add stop guidance after mode guidance (ablation) | MGD³ · ImageNette, IPC 10, ResNetAP-10 | accuracy: 63.8 -> 66.4 | Helped |
| add stop guidance after mode guidance (ablation) | MGD³ · ImageNette, IPC 10, ResNet-18 | accuracy: 62.0 -> 64.4 | Helped |
| add mode guidance to mode discovery (ablation) | MGD³ · ImageNette, IPC 10, ResNetAP-10 | accuracy: 57.1 -> 63.8 | Helped |
| use GMM instead of k-means centroid modes (ablation) | MGD³ · ImageNette, IPC 10 | accuracy: 66.4 -> 66.9 | Helped |
| continue mode guidance throughout the full reverse process instead of stopping early (ablation) | MGD³ · ImageNette | class consistency | Hurt |
| increase the mode guidance scale too much (ablation) | MGD³ · ImageNette, IPC 10 | accuracy | Hurt |
| use soft labels instead of hard labels (ablation) | MGD³ with ResNet-18 · ImageNet-100, IPC 10 | accuracy: 23.6 -> 34.0 | Helped |
| use soft labels instead of hard labels (ablation) | MGD³ with ResNet-18 · ImageNet-100, IPC 20 | accuracy: 32.6 -> 50.2 | Helped |
Surprising findings
- Guidance on the full reverse process (t_stop=0) degrades class consistency and image quality, so an intermediate stop point (t_stop=25) works best.
- GMM-based mode discovery slightly outperforms K-means in the appendix ablation (66.9 vs 66.4 on ImageNette IPC10), though K-means is the default.
Limitations
- Performance on ImageNet-100 is not uniformly best; IDC-1 is slightly better at IPC10 with ResNet-18 and ConvNet-6 results trail MinMaxDiff in some settings.
- Best results rely on a diffusion model pre-trained on the target dataset; the general text-to-image setting is weaker.
- Guidance scale and t_stop are hyperparameters tuned on ImageNette, requiring sweeps.
Experimental setup
Datasets: ImageNet-1K ImageNette ImageIDC ImageNet-100 ImageWoof ImageNet-A ImageNet-B ImageNet-C ImageNet-D ImageNet-E
Benchmarks: ImageNette ImageIDC ImageNet-100 ImageNet-1K ImageWoof ImageNet-A to E
Metrics: top-1 accuracy diversity score representativeness score
Baselines: DM MinMaxDiff LDM D4M DiT IDC-1 Herding Random GLaD H-GLaD LM3D SR2L
Builds on: MinMax Diffusion D4M classifier-free guidance DDIM SRe2L
Closest papers by meaning
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.92)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.91)
- Taming Diffusion for Dataset Distillation with High Representativeness (ICML 2025 · similarity 0.91)
- Efficient Dataset Distillation via Minimax Diffusion (CVPR 2024 · similarity 0.90)
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.89)
- Distribution-Aware Data Expansion with Diffusion Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.