Research map / Label-efficient and robust learning / Knowledge distillation
When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates distilling non-transferable learning (NTL) teachers via data-free KD, finds an "OOD trap" that degrades ID knowledge transfer, and proposes ATEsc, which splits synthetic samples by adversarial robustness and distills only fragile (ID-like) samples while pushing away robust (OOD-like) ones.
Problem
Data-free knowledge distillation assumes trustworthy teachers; when the teacher is an NTL model that restricts transferability from an ID domain to an OOD domain, synthetic data drift toward OOD and the student learns misleading OOD knowledge while losing ID accuracy.
Approach
NTL teachers are more adversarially robust on OOD samples than ID samples, so PGD-based robustness can separate synthetic samples into ID-like (fragile) and OOD-like (robust) groups; distill only on fragile samples and use a KL-maximization forgetting term on robust samples.
Key results
- On CIFAR10->STL10 with VGG-13->VGG-11, DFQ+ATEsc raises OAcc from 52.5 to 66.2... (see table: ATEsc improves IAcc and OAcc over DFQ and CKD across settings)
- On the open-set CIFAR10->Digits backdoor/NTL settings, ATEsc drives OOD-label accuracy to near 0% (e.g., 2.5% to 3.7%) while keeping ID accuracy around 44-65%
- On NTL-based backdoor (CIFAR10, BadNets/Blended/Sig triggers), ATEsc reduces student ASR to roughly 2-68% from near 100% for baselines, though at some cost to ID accuracy
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ATEsc to NAYER | NAYER · SVHN → MNIST-M, close-set; VGG-13 → VGG-11 | IAcc: 7.2 ± 0.3 -> 86.1 ± 1.8 | Helped |
| Add ATEsc to NAYER | NAYER · SVHN → MNIST-M, close-set; VGG-13 → VGG-11 | OAcc: 10.6 ± 0.3 -> 33.7 ± 3.6 | Helped |
| Add ATEsc to NAYER | NAYER · SVHN → CIFAR10, open-set; VGG-13 → VGG-11 | IAcc: 6.4 ± 0.0 -> 89.2 ± 1.6 | Helped |
| Add ATEsc to NAYER | NAYER · SVHN → CIFAR10, open-set; VGG-13 → VGG-11 | OLAcc: 100.0 ± 0.0 -> 0.0 ± 0.0 | Helped |
| Add ATEsc to NAYER | NAYER · BadNet(sq), NTL-based backdoor; VGG-13 → VGG-11 | ASR: 100.0 ± 0.0 -> 34.6 ± 47.3 | Helped |
| Remove BN loss from CMI (ablation) | CMI · CIFAR10 → STL; ResNet-34 → ResNet-18 | IAcc: 37.7 ± 3.5 -> 31.1 ± 3.2 | Hurt |
| Remove BN loss from NAYER (ablation) | NAYER · CIFAR10 → Digits; ResNet-34 → ResNet-18 | IAcc: 49.4 ± 2.6 -> 39.1 ± 3.5 | Hurt |
Surprising findings
- Without BN loss, the DFKD student's ID accuracy can drop to random-guess level, showing BN loss is needed for ID transfer yet is the source of OOD drift.
- Regular KD with OOD data included makes the student inherit OOD misleading knowledge, while ID-only KD yields only ID knowledge.
Limitations
- ATEsc sometimes sacrifices ID accuracy, notably in open-set and backdoor settings.
- Evaluated only on small image datasets (Digits, CIFAR10, STL10, SVHN) with VGG and ResNet at 64x64 resolution.
- Raises IP-protection concerns since it gives a data-free inverse attack against NTL models.
Experimental setup
Datasets: SVHN MNIST-M CIFAR10 STL10 Digits (MNIST, USPS, SVHN, MNIST-M, SYN-D)
Benchmarks: SVHN->MNIST-M CIFAR10->STL10 SVHN->CIFAR10 CIFAR10->Digits CIFAR10 NTL backdoor (BadNets, Blended, Sig)
Metrics: ID accuracy (IAcc) OOD accuracy (OAcc) OOD-label accuracy (OLAcc) attack success rate (ASR)
Baselines: ZSKT DFQ CMI NAYER ABD
Builds on: non-transferable learning ZSKT DFQ CMI NAYER
Closest papers by meaning
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.91)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.89)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.87)
- Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.