Research map / Label-efficient and robust learning / Knowledge distillation
Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition
Chuanguang Yang, Zhulin An, Helong Zhou, Fuzhen Zhuang, Yongjun Xu, Qian Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add instance-level contrastive loss ICL (ablation) | L-MCL · CIFAR-100 ResNet-110 | top-1 accuracy (%): 75.29 -> 78.34 | Improved |
| add instance-level contrastive loss ICL (ablation) | L-MCL · CIFAR-100 WRN-40-2 | top-1 accuracy (%): 76.89 -> 79.29 | Improved |
| add conventional contrastive loss VCL (ablation) | L-MCL · CIFAR-100 ResNet-110 | top-1 accuracy (%): 75.29 -> 77.35 | Improved |
| add conventional contrastive loss VCL (ablation) | L-MCL · CIFAR-100 WRN-40-2 | top-1 accuracy (%): 76.89 -> 77.66 | Improved |
| add soft contrastive loss terms to ICL (ablation) | L-MCL · CIFAR-100 ResNet-110 | top-1 accuracy (%): 78.34 -> 78.83 | Improved |
| add all four contrastive terms (ablation) | L-MCL · CIFAR-100 ResNet-110 | top-1 accuracy (%): 78.83 -> 79.05 | Improved |
| add L-MCL to logit-level online KD (ablation) | logit OKD · CIFAR-100 ResNet-110 | top-1 accuracy (%): 77.13 -> 79.48 | Improved |
| add L-MCL to logit-level online KD (ablation) | logit OKD · CIFAR-100 WRN-40-2 | top-1 accuracy (%): 78.64 -> 80.96 | Improved |
| use weighted all-to-all layer matching instead of one-to-one (ablation) | L-MCL · CIFAR-100 ShuffleV2-ResNet-110 (ResNet-110) | top-1 accuracy (%): 77.12 -> 78.85 | Improved |
| use weighted all-to-all instead of all-to-all layer matching (ablation) | L-MCL · CIFAR-100 ShuffleV2-ResNet-110 (ShuffleV2) | top-1 accuracy (%): 75.29 -> 76.46 | Improved |
| replace logit-based online KD with L-MCL | online KD · STL-10 transfer (WRN-40-2) | top-1 accuracy (%): 62.11 -> 67.86 | Improved |
| apply logit-based online KD methods (DML) | DML · STL-10 transfer (WRN-40-2) | top-1 accuracy (%): 62.11 -> 61.02 | Hurt |
Closest papers by meaning
- Mutual Contrastive Learning for Visual Representation Learning (AAAI 2022 · similarity 0.93)
- Wasserstein Contrastive Representation Distillation (CVPR 2021 · similarity 0.89)
- Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning (CVPR 2023 · similarity 0.88)
- Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration (AAAI 2025 · similarity 0.86)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.86)
- Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
