Research map / Label-efficient and robust learning / Knowledge distillation
NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NAYER, a data-free knowledge distillation method that feeds fixed CLIP label-text embeddings through a randomly reinitialized noisy layer (BatchNorm + Linear) into a generator, achieving higher accuracy and 5-15x faster training than prior DFKD methods.
Problem
Data-free knowledge distillation generators start from random noise lacking semantic information, causing slow training (tens of GPU hours) and low-quality or low-diversity synthetic samples.
Approach
Replace random noise input with a constant label-text embedding (computed once by a text encoder and stored in a pool), and relocate randomness to a noisy layer that is reinitialized every iteration; a single noisy layer can serve all K classes jointly (K-to-1).
Key results
- On CIFAR-100 with WRN40-2/WRN16-2, NAYER (E=300) reaches 71.72% student accuracy versus 69.95% for SpaceshipNet, the prior best.
- NAYER (E=100) trains in roughly 2 hours on a single A100 versus about 31 hours for DeepInv, an average speedup of about 15x.
- On Tiny-ImageNet with ResNet-34 teacher, NAYER (E=300) achieves 64.17% versus 64.04% for SpaceshipNet, and on ImageNet ResNet-50 it reports 68.92% (teacher 75.45%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use label-text embeddings instead of random noise or one-hot labels as generator input (ablation) | NAYER · CIFAR10 | accuracy: 90.14 -> 93.48 | Helped |
| Add the noisy layer and reinitialize it each iteration (ablation) | NAYER · CIFAR10 | accuracy: 14.82 -> 93.48 | Helped |
| Use one noisy layer for the whole class batch rather than one per sample (ablation) | NAYER · CIFAR10 | accuracy: 93.42 -> 93.48 | Helped |
| Add BatchNorm to the noisy layer (ablation) | NAYER · CIFAR10 | accuracy: 91.15 -> 93.48 | Helped |
| Use LTE instead of random noise as generator input (ablation) | NAYER · CIFAR10 | averaging convergence time: - -> 8.52 | Helped |
| Use NL(K-to-1) rather than NL(1-to-1) (ablation) | NAYER · CIFAR10 | diversity score: 0.138 -> 0.139 | Helped |
| Use prompt P2 instead of P3 (ablation) | NAYER · CIFAR10 | accuracy: 93.28 -> 94.07 | Helped |
| Use prompt P2 instead of P3 (ablation) | NAYER · CIFAR100 | accuracy: 71.12 -> 71.72 | Helped |
| Replace CLIP with Doc2Vec as text encoder (ablation) | NAYER · CIFAR10 | accuracy: 94.07 -> 93.98 | No clear effect |
| Replace CLIP with SBERT as text encoder (ablation) | NAYER · CIFAR100 | accuracy: 71.72 -> 71.63 | No clear effect |
| Train NAYER for 100 epochs instead of DeepInv | Data-free distillation on CIFAR10 and CIFAR100 · CIFAR10 and CIFAR100 | training time: 31.23h -> 2.05h; 31.23h -> 2.15h | Helped |
Surprising findings
- Using a single noisy layer across all K classes (K-to-1) converges faster and is more diverse than one noisy layer per image.
- Using only the label index instead of class name as prompt still outperforms the best baseline on CIFAR-100 (71.12% vs 69.95%).
Limitations
- Method is described as straightforward, without incorporating techniques from current SOTA approaches such as activation region constraints or feature exchange.
- Does not outperform SpaceshipNet on the ResNet-32/ResNet-18 CIFAR-10 setting.
- Relies on a text encoder and class names that carry meaningful semantics; label-index prompts are offered as a fallback.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet
Metrics: top-1 accuracy training time (GPU hours) averaging convergence time diversity score Inception Score
Baselines: DeepInv DFQ ZSKT CMI PREKD MBDFKD FM MAD KAKR SpaceshipNet ZeroQ ZAQ
Builds on: DeepInv CMI FM (fast meta generator) Conditional GAN ZeroQ
Closest papers by meaning
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.87)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.86)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.86)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.86)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.86)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.