Research map / Label-efficient and robust learning / Knowledge distillation
Improving Self-Supervised Lightweight Model Learning via Hard-Aware Metric Distillation
Hao Liu, Mang Ye
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add teacher-guided weighting factors (ablation) | SMD · CIFAR-100 supervised distillation, MobileNetV2 | Top-1 accuracy: 63.48 -> 70.76 | Improved |
| add teacher-guided weighting factors (ablation) | SMD · CIFAR-100 supervised distillation, resnet20 | Top-1 accuracy: 69.05 -> 71.56 | Improved |
| add L2 embedding alignment in first epochs (ablation) | SMD · CIFAR-100 supervised distillation, MobileNetV2 | Top-1 accuracy: 65.65 -> 70.76 | Improved |
| add L2 embedding alignment in first epochs (ablation) | SMD · CIFAR-100 supervised distillation, resnet20 | Top-1 accuracy: 71.64 vs 71.56 | No clear effect |
| add the SSL loss to SMD (ablation) | SMD · CIFAR-100 self-supervised distillation, resnet32x4 -> MobileNetV2 | Top-1 accuracy: 61.89 vs 61.32 | No clear effect |
| use metric distillation instead of L2 feature alignment | student MobileNetV2 · CIFAR-100 self-supervised distillation, resnet32x4 -> MobileNetV2 | Top-1 accuracy: 58.43 -> 61.89 | Improved |
| distill with SMD from a ResNet-50 teacher instead of plain SimSiam | ResNet-18 · ImageNet linear eval | Top-1 accuracy: 30.5 -> 61.8 | Improved |
| use SMD instead of CompR-1q under the same training strategy | ResNet-18 · ImageNet linear eval | Top-1 accuracy: 60.5 -> 61.8 | Improved |
| use SMD instead of DisCo | EfficientNet-B0 · ImageNet linear eval | Top-1 accuracy: 66.5 vs 66.5 | No clear effect |
| use SMD instead of WCoRD with no label information | MobileNetV2 (ResNet-50 teacher) · CIFAR-100 supervised distillation | Top-1 accuracy: 70.45 -> 70.76 | Improved |
| cut-off at zero weighting lets student pass teacher | WRN-16-2 (WRN-40-2 teacher) · CIFAR-100 supervised distillation | Top-1 accuracy: 75.61 -> 76.39 | Improved |
| use a stronger distillation signal than the student baseline | MobileNetV2 · CIFAR-100 self-supervised distillation | Top-1 accuracy: 35.87 -> 61.89 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- DisCo — DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for Deep Neural Networks (CVPR 2021) · better on 5 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Effective Self-supervised Pre-training on Low-compute Networks without Distillation (ICLR 2023 · similarity 0.88)
- Learning Metrics From Teachers: Compact Networks for Image Embedding (CVPR 2019 · similarity 0.88)
- DisCo: Remedying Self-Supervised Learning on Lightweight Models with Distilled Contrastive Learning (ECCV 2022 · similarity 0.87)
- A Closer Look at Self-Supervised Lightweight Vision Transformers (ICML 2023 · similarity 0.87)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.85)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
