Research map / Label-efficient and robust learning / Knowledge distillation
Paraphrasing Complex Network: Network Compression via Factor Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use paraphraser plus translator factor transfer instead of attention transfer | ResNet-20 student · CIFAR-10 ResNet-56 teacher | error (%): 7.13 -> 6.85 | Helped |
| use factor transfer instead of KD | ResNet-20 student · CIFAR-100 ResNet-110 teacher | error (%): 33.14 -> 29.08 | Helped |
| use factor transfer on ImageNet | ResNet-18 · ImageNet top-1 | error (%): 29.91 -> 28.57 | Helped |
| use factor transfer for Faster-RCNN backbone | VGG-16 · PASCAL VOC 2007 | mAP: 69.5 -> 70.3 | Helped |
| add translator to paraphraser (ablation) | FT · CIFAR-10 | error (%): 6.18 -> 5.71 | Helped |
| add paraphraser to translator (ablation) | FT · CIFAR-100 | error (%): 27.39 -> 26.91 | Helped |
| increase paraphraser layers from 1 to 3 (ablation) | FT · CIFAR-10 | error (%): 6.09 -> 5.71 | Helped |
| use spatial-preserving paraphraser instead of autoencoder (CAE) (ablation) | FT · CIFAR-10 VGG-13 student | error (%): 5.85 -> 4.84 | Helped |
| apply FT loss at groups 1-3 instead of last group only (ablation) | FT ResNet-20 · CIFAR-10 ResNet-56 teacher | error (%): 7.01% (0.06% higher than single FT) | No clear effect |
| combine FT with KD | FT · CIFAR-100 ResNet-56 student | error (%): 25.62 -> 26.93 | Hurt |
| combine FT with KD | FT · CIFAR-10 VGG-13 student | error (%): 4.84 -> 4.65 | Helped |
| use paraphrased factors instead of full activation maps | WRN-16-1 student · CIFAR-10 WRN-40-1 teacher | error (%): 8.39 -> 6.58 | Helped |
Closest papers by meaning
- Hierarchical Knowledge Squeezed Adversarial Network Compression (AAAI 2020 · similarity 0.87)
- Few Sample Knowledge Distillation for Efficient Network Compression (CVPR 2020 · similarity 0.86)
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · similarity 0.86)
- Progressive Network Grafting for Few-Shot Knowledge Distillation (AAAI 2021 · similarity 0.85)
- Variational Information Distillation for Knowledge Transfer (CVPR 2019 · similarity 0.85)
- Robust Knowledge Transfer via Hybrid Forward on the Teacher-Student Model (AAAI 2021 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.