Research map / Image generation / Fast diffusion sampling
Efficient Dataset Distillation via Minimax Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills surrogate datasets by fine-tuning a pre-trained DiT with PEFT (DiffFit) plus minimax memory-based representativeness and diversity losses, generating images for dataset distillation far faster than sample-wise optimization.
Problem
Sample-wise iterative pixel/embedding optimization for dataset distillation becomes prohibitively expensive as images-per-class and resolution grow, and generated diffusion data lacks representativeness and diversity.
Approach
Fine-tune a pre-trained diffusion model (DiT with DiffFit) using two extra minimax objectives over memory banks of real and predicted embeddings: pull predictions toward their least similar real sample (representativeness) and push them away from their most similar generated sample (diversity).
Key results
- On ImageWoof at 100 IPC with ResNet-18, the method reaches 65.7% vs 60.2% for IDC-1 and 58.9% for DiT, with under one-twentieth of the distillation time of previous methods.
- On ImageNet-1K at 50 IPC, the method reaches 58.6% vs 56.5% for RDED and 46.8% for SRe2L.
- Lowest feature MMD on average across IPC 50 and 100 (4.0 and 4.3) compared with DM, IDC-1 and DiT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add minimax representativeness constraint instead of aligning embeddings to the distribution center (ablation) | Minimax Diffusion · ImageWoof, 10 IPC | top-1 accuracy: 34.4±1.1 -> 37.4±0.4 | Helped |
| Add minimax representativeness constraint instead of aligning embeddings to the distribution center (ablation) | Minimax Diffusion · ImageWoof, 50 IPC | top-1 accuracy: 47.1±0.5 -> 49.5±1.0 | Helped |
| Add minimax representativeness constraint instead of aligning embeddings to the distribution center (ablation) | Minimax Diffusion · ImageIDC, 10 IPC | top-1 accuracy: 49.6±0.7 -> 54.5±1.2 | Helped |
| Add minimax representativeness constraint instead of aligning embeddings to the distribution center (ablation) | Minimax Diffusion · ImageIDC, 50 IPC | top-1 accuracy: 60.2±1.2 -> 65.0±0.8 | Helped |
| Add minimax diversity constraint (ablation) | Minimax Diffusion · ImageWoof, 10 IPC | top-1 accuracy: 35.7±0.8 -> 38.7±0.9 | Helped |
| Add minimax diversity constraint (ablation) | Minimax Diffusion · ImageWoof, 50 IPC | top-1 accuracy: 48.3±0.6 -> 54.9±0.7 | Helped |
| Add minimax diversity constraint (ablation) | Minimax Diffusion · ImageIDC, 10 IPC | top-1 accuracy: 51.5±0.6 -> 52.2±0.6 | Helped |
| Add minimax diversity constraint (ablation) | Minimax Diffusion · ImageIDC, 50 IPC | top-1 accuracy: 64.8±0.8 -> 68.4±0.7 | Helped |
| Increase diversity weight λd to 0.03 (ablation) | Minimax Diffusion · ImageWoof, varying IPC | validation accuracy | Hurt |
Surprising findings
- Pre-trained DiT-generated and random original images fall below random original images at large IPC (e.g., 100 IPC on ImageWoof), showing raw generated data lacks diversity.
Limitations
- Evaluated only on object classification with ImageNet subsets
- Diversity weight too large harms representativeness; hyperparameters tuned on ImageWoof at 100 IPC
Experimental setup
Datasets: ImageWoof ImageNette ImageIDC ImageNet-1K
Benchmarks: ImageWoof ImageNette ImageIDC ImageNet-1K
Metrics: top-1 accuracy MMD
Baselines: Random K-Center Herding DiT DM IDC-1 GLaD DiffFit SRe2L RDED
Builds on: DiT DiffFit IDC Distribution Matching (DM) diffusion control (Wang et al.)
Closest papers by meaning
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.92)
- Taming Diffusion for Dataset Distillation with High Representativeness (ICML 2025 · similarity 0.92)
- Data-free Distillation of Diffusion Models with Bootstrapping (ICML 2024 · similarity 0.91)
- MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models (ICML 2025 · similarity 0.90)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.90)
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.