Research map / Image generation / Fast diffusion sampling
Taming Diffusion for Dataset Distillation with High Representativeness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D$^3$HR, a diffusion-based dataset distillation framework that uses DDIM inversion to map VAE latents to a higher-normality Gaussian domain and group sampling to select the most representative latent subset.
Problem
Diffusion-based dataset distillation struggles to produce representative distilled images because VAE latent distributions are complex non-Gaussian mixtures, random initial noise causes distribution deviation, and latents are sampled individually rather than as a set.
Approach
Apply deterministic DDIM inversion to map class latents into a near-Gaussian noise domain, fit a Gaussian by its mean and variance, then sample many candidate subsets and pick the one whose mean, std and skewness best match the fitted Gaussian before DDIM sampling and decoding.
Key results
- On ImageNet-1K IPC=10 with ResNet-18, D3HR reaches 44.3% top-1 versus 42.0% for RDED and 37.9% for DWA.
- On CIFAR-10 IPC=50 with ResNet-18, D3HR reaches 70.8% versus 62.1% for RDED, a reported 12.5% relative improvement.
- DDIM inversion domain mapping outperforms DDPM-based mapping by 11.5% accuracy in the ablation (41.6 vs 37.3 base variants on ImageNet-1K IPC=10 ResNet-18 in the table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDIM inversion instead of DDPM domain mapping (ablation) | D³HR without group sampling · ImageNet-1K, IPC 10, ResNet-18 | Top-1 accuracy: 37.3\pm0.7 -> 41.6\pm0.2 | Helped |
| Select a representative group of Gaussian samples rather than sampling individually (ablation) | D³HR · ImageNet-1K, IPC 10, ResNet-18 | Top-1 accuracy: 41.6\pm0.2 -> 44.3\pm0.3 | Helped |
| Include mean-matching loss in group sampling (ablation) | D³HR · ImageNet-1K, IPC 10, ResNet-18 | Top-1 accuracy: 41.6\pm0.2 -> 42.6\pm0.1 | Helped |
| Include standard-deviation matching loss in group sampling (ablation) | D³HR · ImageNet-1K, IPC 10, ResNet-18 | Top-1 accuracy: 41.6\pm0.2 -> 42.3\pm0.2 | Helped |
| Include skewness-matching loss in group sampling (ablation) | D³HR · ImageNet-1K, IPC 10, ResNet-18 | Top-1 accuracy: 41.6\pm0.2 -> 42.4\pm0.1 | Helped |
| Use all three sampling losses rather than any two (ablation) | D³HR · ImageNet-1K, IPC 10, ResNet-18 | Top-1 accuracy: 43.3\pm0.2 -> 44.3\pm0.3 | Helped |
| Increase inversion steps from 20 to 31, then to 40 (ablation) | D³HR · ImageNet-1K, IPC 10 | Top-1 accuracy | Mixed |
| Increase the number of candidate subsets (ablation) | D³HR group sampling · Tiny-ImageNet, IPC 10 | Accuracy: 40.2\pm0.4 -> 44.2\pm0.2 | Helped |
| Increase the number of candidate subsets (ablation) | D³HR group sampling · CIFAR-10, IPC 10 | Accuracy: 38.9\pm0.3 -> 42.1\pm0.2 | Helped |
| Change the sampling distribution variance from its default (ablation) | D³HR group sampling · ImageNet-1K, IPC 10 | Accuracy: 44.1\pm0.3 -> 39.8\pm0.5; 44.1\pm0.3 -> 34.0\pm0.6 | Hurt |
| Change the relative weights of mean, standard deviation, and skewness losses (ablation) | D³HR group sampling · ImageNet-1K, IPC 10 | Accuracy: 44.2\pm0.1; 44.0\pm0.4; 43.6\pm0.3; 43.4\pm0.2 | No clear effect |
| Use five teacher soft labels instead of one (ablation) | D³HR validation · ImageNet-1K | Top-1 accuracy | Helped |
Surprising findings
- Increasing or decreasing the sampling variance by 10-50% reduces accuracy, so matching the fitted distribution exactly works best.
- Performance peaks at 31 DDIM inversion steps; larger step counts degrade reconstruction despite better Gaussianity.
Limitations
- Relies on a DiT pretrained on ImageNet-1K that must be fine-tuned per dataset (400 epochs for CIFAR and Tiny-ImageNet).
- Single-Gaussian matching cannot capture mixture structure at small inversion steps, so performance depends on choosing the inversion step count (t=31 best).
- Evaluation uses soft labels from a teacher, so the comparison to hard-label settings relies on separate experiments.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K ImageWoof
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K ImageWoof
Metrics: top-1 accuracy
Baselines: SRe2L DWA D4M RDED D3S Minimax G-VBSM ESFR HMDC Teddy
Builds on: DDIM D4M Minimax SRe2L RDED
Closest papers by meaning
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.93)
- Efficient Dataset Distillation via Minimax Diffusion (CVPR 2024 · similarity 0.92)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.91)
- MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models (ICML 2025 · similarity 0.91)
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.90)
- Data-free Distillation of Diffusion Models with Bootstrapping (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.