Research map / Label-efficient and robust learning / Knowledge distillation
Long Short-Term Sample Distillation
Liang Jiang, Zujie Wen, Zhongping Liang, Yafang Wang, Gerard de Melo, Zhe Li, Liangzhuang Ma, Jiaxing Zhang, Xiaolong Li, Yuan Qi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use long-term and short-term sample-specific teachers together | LSTSD · CIFAR100 | accuracy (%): 69.42 vs. 68.37; 71.51 vs. 70.26; 73.17 vs. 72.00; 73.83 vs. 72.57; 79.35 vs. 78.83 | Improved |
| use long-term and short-term sample-specific teachers together | LSTSD with BERT · RTE, MRPC, SST-2 | accuracy (%): 74.73 vs. 74.01; 89.22 vs. 87.25; 93.35 vs. 93.12 | Improved |
| use long-term and short-term sample-specific teachers together | LSTSD with BERT · CoLA | Matthews correlation (%): 61.59 vs. 61.75 | No clear effect |
| use long-term and short-term sample-specific teachers together | LSTSD with CNN · RTE, SST-2, CoLA | accuracy or Matthews correlation (%): 57.40 vs. 56.68; 72.36 vs. 71.67; 14.50 vs. 12.81 | Improved |
| use long-term and short-term sample-specific teachers together | LSTSD with CNN · MRPC | accuracy (%): 73.28 vs. 73.77 | Hurt |
| add the long-term teacher (ablation) | LSTSD · CIFAR100, ResNet-20 | accuracy (%): 69.09 -> 69.42 | Improved |
| add the short-term teacher (ablation) | LSTSD · CIFAR100, ResNet-20 | accuracy (%): 68.82 -> 69.42 | Improved |
| use numerous sample-specific teachers instead of shared teachers (ablation) | LSTSD · CIFAR100, ResNet-20 | accuracy (%): 67.85 -> 69.42 | Improved |
| increase mini-generation length from 1 to 6 epochs (ablation) | LSTSD · CIFAR100, ResNet-20 | accuracy | Improved |
| increase mini-generation length beyond 6 epochs toward 40 (ablation) | LSTSD · CIFAR100, ResNet-20 | accuracy | Hurt |
Closest papers by meaning
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.88)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.88)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.87)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.87)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
