Research map / Label-efficient and robust learning / Knowledge distillation
Online Knowledge Distillation via Collaborative Learning
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train ResNet-50 and ResNet-18 collaboratively with KDCL-Linear soft targets | ResNet-50 and ResNet-18 · ImageNet-2012 val | Top-1 accuracy (%): 76.8 -> 77.8; 71.2 -> 73.1 | Helped |
| Train ResNet-50 and ResNet-18 with KDCL using three compact peer models | ResNet-50 and ResNet-18 · ImageNet | Top-1 accuracy (%): 76.8 -> 78.2; 71.2 -> 73.5 | Helped |
| Collaboratively train MobileNetV2 and MobileNetV2x0.5 with KDCL | MobileNetV2 and MobileNetV2x0.5 · ImageNet | Top-1 accuracy (%): 72.0 -> 74.0; 64.8 -> 66.9 | Helped |
| Use KDCL-pretrained ResNet-18 rather than baseline-pretrained ResNet-18 | Faster R-CNN with ResNet-18 backbone · COCO 2017 val (minival) | box mAP: 32.2 -> 33.1 | Helped |
| Use KDCL-pretrained ResNet-18 rather than baseline-pretrained ResNet-18 | Mask R-CNN with ResNet-18 backbone · COCO 2017 val (minival) | box mAP: 33.4 -> 34.0 | Helped |
| Use KDCL-pretrained ResNet-18 rather than baseline-pretrained ResNet-18 | Mask R-CNN with ResNet-18 backbone · COCO 2017 val (minival) | mask mAP: 30.7 -> 31.3 | Helped |
| Add KDCL collaborative distillation | ResNet-32 and WRN-16-2 · CIFAR-100 | Accuracy (%): 69.9 -> 74.3; 72.2 -> 75.5 | Helped |
| Add validation-set-weighted generalization-based soft targets instead of KDCL-Naive (ablation) | ResNet-32 and WRN-16-2 · CIFAR-100 | Accuracy (%): 73.7 -> 74.3; 74.8 -> 75.5 | Helped |
| Add invariant collaborative learning with separately distorted inputs (ablation) | KDCL-General on ResNet-32 and WRN-16-2 · CIFAR-100 | Accuracy (%): 74.0 -> 74.3; 75.2 -> 75.5 | Helped |
| Replace KDCL-Naive with KDCL-Linear soft targets (ablation) | ResNet-32 and WRN-16-2 · CIFAR-100 | Accuracy (%): 73.7 -> 73.4; 74.8 -> 74.6 | Mixed |
| Replace KDCL-Naive with KDCL-MinLogit soft targets (ablation) | ResNet-32 and WRN-16-2 · CIFAR-100 | Accuracy (%): 73.7 -> 73.0; 74.8 -> 74.1 | Hurt |
| Use validation-weighted KDCL-General rather than KDCL-Linear or KDCL-MinLogit (ablation) | ResNet-50 and ResNet-18 · ImageNet-2012 val | Top-1 accuracy (%): 77.8 -> 77.1; 73.1 -> 72.0 | Hurt |
Closest papers by meaning
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.87)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.87)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.87)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.87)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.87)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.