Research map / Label-efficient and robust learning / Knowledge distillation
Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning
Kaiyou Song, Jin Xie, Shan Zhang, Zimeng Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train R50 and ViT-B collaboratively with MOKD instead of independently | R50 and ViT-B · ImageNet val | linear probing accuracy: 72.1 -> 75.6 (R50); 77.0 -> 78.0 (ViT-B) | Improved |
| Train R50 and ViT-B collaboratively with MOKD instead of independently | R50 and ViT-B · ImageNet val | k-NN accuracy: 62.8 -> 70.6 (R50); 73.6 -> 75.2 (ViT-B) | Improved |
| Use MOKD rather than the strongest listed SSL-KD baseline DisCo | R34 with R50 teacher · ImageNet val | linear probing accuracy: 62.5 -> 67.0 | Improved |
| Use MOKD rather than independent pre-training | R50 with ViT-B · ImageNet 1% subset | top-1 accuracy: 49.5 -> 57.2 | Improved |
| Use MOKD rather than independent pre-training | R50 with ViT-S · CIFAR-100 | top-1 accuracy: 85.3 -> 86.2 | Improved |
| Use MOKD rather than DisCo pre-training | R34 with R50 teacher · COCO val2017 | detection APᵇ: 40.0 -> 40.3 | Improved |
| Add cross-distillation L_cm to self-distillation L_sm (ablation) | MOKD, R50 and ViT-S · ImageNet100 val | linear probing top-1 accuracy: 87.0 -> 87.2 (R50); 80.3 -> 82.6 (ViT-S) | Improved |
| Add T-Head self-distillation L_st (ablation) | MOKD, R50 and ViT-S · ImageNet100 val | linear probing top-1 accuracy: 87.2 -> 87.7 (R50); 82.6 -> 83.5 (ViT-S) | Improved |
| Add T-Head cross-distillation L_ct (ablation) | MOKD, R50 and ViT-S · ImageNet100 val | linear probing top-1 accuracy: 87.7 -> 88.3 (R50); 83.5 -> 84.6 (ViT-S) | Improved |
| Increase T-Head depth from one to three transformer blocks (ablation) | MOKD, R50 and ViT-S · ImageNet100 val | linear probing top-1 accuracy: 87.4, 83.3 -> 88.3, 84.6 | Improved |
| Increase cross-distillation weight for the stronger R50 while keeping ViT-S weight at 1 (ablation) | MOKD, R50-ViT-S · ImageNet100 val | linear probing top-1 accuracy: 88.3 -> 87.4 (R50); 84.6 -> 84.1 (ViT-S) | Hurt |
| Increase cross-distillation weight for the weaker ViT-S while keeping R50 weight at 0.1 (ablation) | MOKD, R50-ViT-S · ImageNet100 val | linear probing top-1 accuracy: 80.4 -> 84.6 (ViT-S) | Improved |
Closest papers by meaning
- A Fast Knowledge Distillation Framework for Visual Recognition (ECCV 2022 · similarity 0.89)
- Multi-target Knowledge Distillation via Student Self-reflection (IJCV 2023 · similarity 0.89)
- Multi-label Self Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition (TPAMI 2023 · similarity 0.88)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.86)
- Resource-Aware Federated Self-Supervised Learning with Global Class Representations (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
