Research map / Label-efficient and robust learning / Knowledge distillation
Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multivariate-normal FC-2 features instead of softmax-space modeling (ablation) | Data-Free KD with Soft Targeted Transfer Set Synthesis · MNIST | accuracy: 98.92% -> 99.04% | Helped |
| Add the activation loss while modeling the softmax space (ablation) | Data-Free KD with Soft Targeted Transfer Set Synthesis · MNIST | accuracy: 98.92% -> 98.96% | Helped |
| Add activation loss to FC-2 feature-space modeling (ablation) | Data-Free KD with Soft Targeted Transfer Set Synthesis · MNIST | accuracy: 99.04% -> 99.08% | Helped |
| Use multivariate-normal FC-2 features instead of softmax-space modeling (ablation) | Data-Free KD with Soft Targeted Transfer Set Synthesis · CIFAR-10 | accuracy: 71.95% -> 73.54% | Helped |
| Add the activation loss while modeling the softmax space (ablation) | Data-Free KD with Soft Targeted Transfer Set Synthesis · CIFAR-10 | accuracy: 71.95% -> 72.46% | Helped |
| Add activation loss to FC-2 feature-space modeling (ablation) | Data-Free KD with Soft Targeted Transfer Set Synthesis · CIFAR-10 | accuracy: 73.54% -> 73.91% | Helped |
| Vary the feature-distribution variance σ across the tested range (ablation) | Data-Free KD with Soft Targeted Transfer Set Synthesis · MNIST and CIFAR-10 | accuracy | Mixed |
| Vary activation-loss weight λa away from 0.05 (ablation) | Data-Free KD with Soft Targeted Transfer Set Synthesis · MNIST and CIFAR-10 | accuracy | Hurt |
| Optimize noise inputs toward teacher-generated soft targets rather than using unoptimized noise | Data-Free KD with Soft Targeted Transfer Set Synthesis · MNIST | accuracy: 87.58% -> 99.08% | Helped |
| Use the proposed synthesized transfer set rather than unoptimized noise inputs | Data-Free KD with Soft Targeted Transfer Set Synthesis · CIFAR-10 | accuracy: 36.49% -> 73.91% | Helped |
| Use the proposed synthesized transfer set rather than unoptimized noise inputs | Data-Free KD with Soft Targeted Transfer Set Synthesis · CIFAR-10 | accuracy: 19.49% -> 93.31% | Helped |
Closest papers by meaning
- Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model (ICML 2021 · similarity 0.92)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.87)
- What Makes a Good Dataset for Knowledge Distillation? (CVPR 2025 · similarity 0.87)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.