Research map / Image generation / Text-to-image diffusion
When Model Knowledge meets Diffusion Model: Diffusion-assisted Data-free Image Synthesis with Alignment of Domain and Class
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDIS, which uses frozen Stable Diffusion 2.1 as an image prior for data-free image synthesis, aligning domain via BatchNorm-statistics guidance during sampling and class via optimizing a single class alignment token embedding.
Problem
Data-free image synthesis (DFIS) from pre-trained classifiers produces images that deviate from the original training distribution because the search space is huge and lacks natural-image priors; naive text-to-image prompts are too vague to match training domain and class details.
Approach
Use a frozen text-to-image diffusion model as an image prior, steer sampling with a BatchNorm-running-statistics loss (Domain Alignment Guidance) so generated images match the classifier's training domain, and optimize one pseudo-word token embedding (Class Alignment Token) with cross-entropy on the classifier to capture class-specific details.
Key results
- On ImageNet-1k, DDIS reaches IS 15.92 and FID 30.31 versus DeepInversion's IS 9.52 and FID 187.63.
- On PACS Art painting, the ablation shows FID drops from 193.57 (vanilla SD) to 133.37 with both DAG and CAT.
- In data-free KD on ImageNet-1k (ResNet-34 to ResNet-18), DDIS gives 41.68 top-1 versus 43.30 for original data, 33.02 for SD, 4.67 for DI and 2.01 for PII.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDIS instead of DeepInversion or PlugInInversion | Image synthesis with Stable Diffusion 2.1 · ImageNet-1k Photo; PACS Art painting and Cartoon; Style-Aligned Caricature and Manga | FID: 187.63 -> 30.31; 188.53 -> 133.37; 148.94 -> 85.41; 195.25 -> 139.75; 206.57 -> 145.82 | Helped |
| Use DDIS-generated images instead of Stable Diffusion images for distillation | Student networks on PACS-art · PACS-art | accuracy: 21.93 -> 28.46; 18.12 -> 21.89; 31.67 -> 35.87; 19.52 -> 24.65; 22.61 -> 27.43; 41.74 -> 48.56 | Helped |
| Use DDIS-generated images instead of Stable Diffusion images for distillation | Student networks on PACS-cartoon · PACS-cartoon | accuracy: 38.12 -> 47.06; 38.94 -> 44.92; 44.67 -> 51.45; 36.73 -> 42.93; 38.15 -> 44.81; 43.63 -> 49.32 | Helped |
| Add DAG to Stable Diffusion sampling (ablation) | DDIS image synthesis · PACS Art painting | FID: 193.57 -> 174.31 | Helped |
| Add CAT to Stable Diffusion sampling (ablation) | DDIS image synthesis · PACS Art painting | FID: 193.57 -> 166.22 | Helped |
| Combine DAG and CAT (ablation) | DDIS image synthesis · PACS Art painting | FID: 193.57 -> 133.37 | Helped |
| Fine-tune the SD UNet rather than optimize CAT with frozen SD (ablation) | Stable Diffusion 2.1 · ImageNet-1k ambiguous classes | Avg. Confidence: 78.36 -> 1.36 | Hurt |
| Fine-tune the SD text encoder rather than optimize CAT with frozen SD (ablation) | Stable Diffusion 2.1 · ImageNet-1k ambiguous classes | Avg. Confidence: 78.36 -> 10.30 | Hurt |
| Fine-tune the full SD model rather than optimize CAT with frozen SD (ablation) | Stable Diffusion 2.1 · ImageNet-1k ambiguous classes | Avg. Confidence: 78.36 -> 0.17 | Hurt |
| Optimize multiple CAT tokens instead of a single token (ablation) | Frozen Stable Diffusion 2.1 · ImageNet-1k ambiguous classes | Avg. Confidence: 78.36 -> 9.83; 9.60; 9.71; 8.50 | Hurt |
| Add BatchNorm loss to CAT embedding optimization (ablation) | Frozen Stable Diffusion 2.1 · ImageNet-1k ambiguous classes | Avg. Confidence: 78.36 -> 8.61 | Hurt |
Surprising findings
- Fine-tuning Stable Diffusion (UNet, text encoder, or full) in DFIS produces degraded, low-confidence images, whereas optimizing only one token on frozen SD works.
- Adding the BatchNorm loss to CAT optimization hurts class separability, producing mixed-class images.
Limitations
- Fails on the Sketch domain due to abstract representations.
- DAG relies on BatchNorm statistics and only applies to models with BN layers.
Experimental setup
Datasets: ImageNet-1k PACS Style-Aligned
Benchmarks: ImageNet-1k PACS Style-Aligned
Metrics: IS FID Precision Recall top-1 accuracy of distilled student pruned model accuracy
Baselines: DeepInversion PlugInInversion Stable Diffusion 2.1 (vanilla)
Builds on: DeepInversion Plug-In-Inversion Stable Diffusion classifier-free guidance textual inversion
Closest papers by meaning
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.90)
- Image is All You Need to Empower Large-scale Diffusion Models for In-Domain Generation (CVPR 2025 · similarity 0.89)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.89)
- Aligned Datasets Improve Detection of Latent Diffusion-Generated Images (ICLR 2025 · similarity 0.88)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.88)
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.