Research map / Label-efficient and robust learning / Knowledge distillation
Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MBD-based robustness labels instead of standard KD softmax targets | DB3KD student training · MNIST, Fashion-MNIST, CIFAR-10 | accuracy: 99.33% -> 99.52%; 90.82% -> 91.45%; 77.81% -> 78.30% | Helped |
| Use MBD-based robustness labels instead of standard KD softmax targets | DB3KD student training · FLOWERS102 | accuracy: 94.05% -> 93.77% | Hurt |
| Replace SD robustness with BD boundary distances (ablation) | DB3KD · MNIST, LeNet5-half | accuracy: 99.15% -> 99.51% | Helped |
| Replace BD distances with optimized MBD distances (ablation) | DB3KD · MNIST, LeNet5-half | accuracy: 99.51% -> 99.52% | No clear effect |
| Use DB3KD rather than a same-size surrogate teacher | Student distillation · FLOWERS102 | accuracy: 92.93% -> 93.77% | Helped |
| Use DB3KD with ResNet-50 rather than ResNet-34 teacher | CIFAR-100 ResNet-18 student · CIFAR-100 | accuracy: 77.31% -> 78.65% | Helped |
| Generate pseudo inputs by moving them away from teacher decision boundaries rather than using random noise directly | ZSDB3KD · CIFAR-10 | accuracy: 13.53% -> 59.46% | Helped |
| Add data augmentation to the generated transfer set (ablation) | ZSDB3KD · MNIST | accuracy: 94.20% -> 96.54% | Helped |
| Add data augmentation to the generated transfer set (ablation) | ZSDB3KD · Fashion-MNIST | accuracy: 67.24% -> 72.31% | Helped |
| Add data augmentation to the generated transfer set (ablation) | ZSDB3KD · CIFAR-10 | accuracy: 37.58% -> 59.46% | Helped |
| Increase pseudo-sample generation iterations, pushing samples farther from boundaries (ablation) | ZSDB3KD · MNIST | accuracy | Helped |
| Increase generated transfer samples from 1k to 8k per category (ablation) | ZSDB3KD · Fashion-MNIST | accuracy: 61.39% to 67.24% | Helped |
Closest papers by meaning
- Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis (AAAI 2021 · similarity 0.92)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.90)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.89)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.88)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.88)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.