Research map / Label-efficient and robust learning / Knowledge distillation
Lipschitz Continuity Guided Knowledge Distillation
Yuzhang Shang, Bin Duan, Ziliang Zong, Liqiang Nie, Yan Yan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Lipschitz continuity loss (lambda 0 -> 3.2) (ablation) | KD, setup (a) · CIFAR-100 setup (a) | top-1 error (%): 21.69 -> 20.33 | Improved |
| add Lipschitz continuity loss (lambda 0 -> 3.2) (ablation) | KD, setup (b) · CIFAR-100 setup (b) | top-1 error (%): 23.43 -> 21.48 | Improved |
| add Lipschitz continuity loss (lambda 0 -> 1.6) (ablation) | KD, setup (c) · CIFAR-100 setup (c) | top-1 error (%): 26.47 -> 23.56 | Improved |
| add Lipschitz continuity loss (lambda 0 -> 3.2) (ablation) | KD, setup (d) · CIFAR-100 setup (d) | top-1 error (%): 27.68 -> 23.78 | Improved |
| raise lambda from 3.2 to 6.4 (ablation) | LONDON, setup (a) · CIFAR-100 setup (a) | top-1 error (%): 20.33 -> 21.87 | Hurt |
| raise lambda from 3.2 to 6.4 (ablation) | LONDON, setup (d) · CIFAR-100 setup (d) | top-1 error (%): 23.78 -> 25.22 | Hurt |
| turn off Lipschitz continuity loss on setup (b) (ablation) | LONDON · CIFAR-100 setup (b) | validation accuracy | Improved |
| use LONDON instead of OFD | OFD, setup (a) · CIFAR-100 setup (a) | top-1 error (%): 20.89 -> 20.33 | Improved |
| use LONDON instead of OFD | OFD, setup (d) · CIFAR-100 setup (d) | top-1 error (%): 24.44 -> 23.78 | Improved |
| use LONDON instead of OFD | OFD · ImageNet val (ResNet152 -> ResNet50) | top-1 error (%): 21.65 -> 21.12 | Improved |
| use LONDON instead of OFD | OFD · ImageNet val (ResNet50 -> MobileNet) | top-1 error (%): 28.75 -> 27.64 | Improved |
| use LONDON instead of OFD | OFD · PASCAL VOC2007 test (SSD, ResNet18) | mAP: 73.08 -> 73.82 | Improved |
Closest papers by meaning
- Achieving Domain-Independent Certified Robustness via Knowledge Continuity (NeurIPS 2024 · similarity 0.89)
- Similarity-Preserving Knowledge Distillation (ICCV 2019 · similarity 0.89)
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks (ICLR 2024 · similarity 0.88)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.87)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.86)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
