Research map / Architectures and efficiency / Pruning and efficient tuning
Few Sample Knowledge Distillation for Efficient Network Compression
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply few-sample FSKD rather than FitNet to a pruned VGG-16 | VGG-16 · CIFAR-10, Prune-A/B/C, 500 samples | accuracy: 92.13±0.35 -> 92.46±0.15; 90.68±0.47 -> 91.21±0.23; 88.31±0.70 -> 90.41±0.31 | Helped |
| Apply FSKD rather than FitNet to network-slimmed models | VGG-19, ResNet-164, and DenseNet-40 · CIFAR-10 and CIFAR-100 | accuracy: 90.47±0.57 -> 93.41±0.23; 88.94±0.66 -> 94.19±0.21; 91.37±0.17 -> 93.62±0.09; 69.52±0.43 -> 71.98±0.15; 73.87±0.45 -> 76 | Helped |
| Apply FSKD rather than FitNet to a network-slimmed VGG-A | VGG-A · ImageNet, 1000 samples | accuracy: 58.3±0.4 -> 62.5±0.2 | Helped |
| Use block-coordinate descent instead of SGD to optimize FSKD (ablation) | Prune-B VGG-16 · CIFAR-10, 100 and 500 samples | accuracy: 89.41 -> 90.17; 90.76 -> 91.21 | Helped |
| Use more than one BCD iteration (ablation) | FSKD · CIFAR-10, Prune-B | accuracy | No clear effect |
| Use samples from CIFAR-100 instead of CIFAR-10 for FSKD (ablation) | Prune-B VGG-16 · CIFAR-10 | accuracy: 90.17±0.31 vs 90.15±0.31; 91.21±0.23 vs 91.20±0.25 | No clear effect |
| Train only pointwise layers while keeping depthwise channels orthogonal rather than training the full network (ablation) | VGG-16 and ResNet-50 · CIFAR-10 and CIFAR-100 | accuracy: 93.00 -> 93.91; 73.35 -> 73.61; 92.64 -> 93.51; 69.93 -> 70.83 | Helped |
| Use FSKD rather than FitNet for few-sample distillation of a randomly initialized student | Hand-designed student networks · MNIST, 10–200 labeled samples | accuracy: 86.1 -> 94.4; 92.3 -> 96.5; 94.5 -> 97.0; 96.0 -> 97.5; 96.5 -> 97.8; 90.3 -> 95.5; 94.2 -> 97.2; 96.1 -> 97.6; 96.7 -> | Helped |
Closest papers by meaning
- Progressive Network Grafting for Few-Shot Knowledge Distillation (AAAI 2021 · similarity 0.91)
- Few Shot Network Compression via Cross Distillation (AAAI 2020 · similarity 0.91)
- EPSD: Early Pruning with Self-Distillation for Efficient Model Compression (AAAI 2024 · similarity 0.90)
- Knowledge distillation via softmax regression representation learning (ICLR 2021 · similarity 0.90)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.