Research map / Image generation / Fast diffusion sampling

DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture

CVPR 2025 · Image generation › Fast diffusion sampling · approach: Fast diffusion ODE solvers · 12 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DKDM, a data-free distillation scheme that trains a new diffusion model of any architecture from a pretrained teacher DM using a dynamic iterative distillation process that produces noisy training samples on the fly.

Problem

Training diffusion models requires massive datasets, and existing diffusion distillation methods usually need training data or copy the teacher's architecture. The paper asks whether a new diffusion model can be trained using only a pretrained diffusion model as data source.

Approach

A DKDM objective replaces the data-dependent diffusion posterior and prior with teacher-generated noisy samples and teacher denoising outputs, and dynamic iterative distillation (shuffled denoising over an enlarged batch set with random selection and substitution) efficiently supplies these samples without full generative chains.

Key results

• On CIFAR10 the dynamic iterative distillation student reaches FID 9.56 versus 12.06 for data-free training and 7.84 for data-based training.
• On CelebA-HQ 256 the student reaches FID 8.69, beating data-based training (9.09) and all data-limited/data-free baselines (14.49 to 15.36).
• Cross-architecture CNN-to-ViT distillation on CIFAR10 gives FID 13.17 versus 44.62 for data-free training.

Measured effects reported in the paper

Surprising findings

• The distilled student outperforms the data-based trained model on some metrics, e.g. IS on CIFAR10 and FID on CelebA-HQ.
• Students trained with the proposed method beat the data-free baselines even though the baselines use full-size synthetic datasets.

Limitations

• Teachers and students are all CNN-based in the main experiments; ViT results only in a cross-architecture CIFAR10 test.
• Evaluation is limited to small-resolution pixel-space and latent-space unconditional/class-free benchmarks.

Experimental setup

Datasets: CIFAR10 CelebA ImageNet 32x32 CelebA-HQ 256 FFHQ 256

Benchmarks: CIFAR10 32x32 CelebA 64x64 ImageNet 32x32 CelebA-HQ 256 FFHQ 256

Metrics: FID sFID IS

Baselines: Data-Free Training Data-Limited Training Data-Based Training

Builds on: DDPM Improved DDPM DDIM Knowledge distillation for diffusion models (BOOT)

Closest papers by meaning

• Data-free Distillation of Diffusion Models with Bootstrapping (ICML 2024 · similarity 0.93)
• When Model Knowledge meets Diffusion Model: Diffusion-assisted Data-free Image Synthesis with Alignment of Domain and Class (ICML 2025 · similarity 0.90)
• Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.90)
• Efficient Dataset Distillation via Minimax Diffusion (CVPR 2024 · similarity 0.90)
• Taming Diffusion for Dataset Distillation with High Representativeness (ICML 2025 · similarity 0.90)
• Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.90)