Research map / Image generation / Fast diffusion sampling
D^4: Dataset Distillation via Disentangled Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D4M, a dataset distillation framework that uses prototypes from a pretrained latent diffusion model's encoder plus label-text conditioning to generate architecture-free synthetic images, with soft-label training-time matching.
Problem
Dataset distillation relies on bi-level optimization tied to a matching architecture, which is computationally expensive on large-scale high-resolution datasets and generalizes poorly across architectures.
Approach
Disentangle feature aggregation from generation: cluster latent encodings of real images into per-class prototypes with Mini-Batch k-Means, use them as noised inputs to a pretrained latent diffusion model conditioned on label text, and defer matching to training time using soft labels from a teacher network, avoiding synthesis-time matching.
Key results
- On ImageNet-1K with IPC-100, D4M reaches 66.5% Top-1 with ResNet-101 student versus 65.9% for SRe2L at IPC-200 (SRe2L 65.8% at IPC-100 with ResNet-101).
- On CIFAR-100 with IPC-10, D4M attains 45.0% Top-1, surpassing FRePO and TESLA by 2.5% and 3.3%.
- Synthesis time on ImageNet-1K is 2.7 s per image at 512x512 with 6.1 GB GPU, versus 5.2 s and 34.8 GB for SRe2L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace STM with TTM (ablation) | D4M · ImageNet-1K, IPC 10, teacher ResNet-18 | Top-1 accuracy: 23.6 -> 27.9(+4.3) | Helped |
| replace STM with TTM (ablation) | D4M · ImageNet-1K, IPC 10, teacher ResNet-50 | Top-1 accuracy: 15.8 -> 20.7(+4.9) | Helped |
| replace STM with TTM (ablation) | D4M · ImageNet-1K, IPC 10, teacher ResNet-101 | Top-1 accuracy: 12.5 -> 19.4(+6.9) | Helped |
| add prototype initialization to the diffusion process (ablation) | D4M · ImageNet-1K, IPC 10 | Top-1 accuracy: 15.6 -> 27.9(+12.3) | Helped |
| add prototype initialization to the diffusion process (ablation) | D4M · Tiny-ImageNet, IPC 10 | Top-1 accuracy: 30.5 -> 46.2(+15.7) | Helped |
| distill 200 Tiny-ImageNet categories from distilled ImageNet-1K (D4M-G) | D4M · Tiny-ImageNet, IPC 50 | Top-1 accuracy: 46.2 -> 46.8 | Helped |
| distill 200 Tiny-ImageNet categories from distilled ImageNet-1K (D4M-G) | D4M · Tiny-ImageNet, IPC 100 | Top-1 accuracy: 51.4 -> 53.3 | Mixed |
Surprising findings
- Synthesis-time matching performed on D4M images degrades accuracy (e.g., ResNet-18 teacher IPC-10 drops from 27.9% to 23.6%), contrary to the expectation that additional matching helps.
Limitations
- Significant performance degradation under extreme distillation (IPC-1 and IPC-10).
- Evaluated only on image classification datasets; plans to distill more real-world multi-modal datasets.
- Relies on soft labels from a teacher network during training, so matching is not fully architecture-free.
Experimental setup
Datasets: ImageNet-1K Tiny-ImageNet CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K Tiny-ImageNet
Metrics: Top-1 accuracy synthesis time GPU memory
Baselines: KIP FRePO DSA CAFE TESLA SRe2L MTT
Builds on: Latent Diffusion Model SRe2L Matching Training Trajectories FKD Mini-Batch k-Means
Closest papers by meaning
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.92)
- MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models (ICML 2025 · similarity 0.91)
- Taming Diffusion for Dataset Distillation with High Representativeness (ICML 2025 · similarity 0.91)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.90)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.90)
- Efficient Dataset Distillation via Minimax Diffusion (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.