AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning

Kaiyou Song, Jin Xie, Shan Zhang, Zimeng Luo

CVPR 2023 · Label-efficient and robust learning › Knowledge distillation · approach: Few-shot and metric distillation · 40 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Train R50 and ViT-B collaboratively with MOKD instead of independentlyR50 and ViT-B · ImageNet vallinear probing accuracy: 72.1 -> 75.6 (R50); 77.0 -> 78.0 (ViT-B)Improved
Train R50 and ViT-B collaboratively with MOKD instead of independentlyR50 and ViT-B · ImageNet valk-NN accuracy: 62.8 -> 70.6 (R50); 73.6 -> 75.2 (ViT-B)Improved
Use MOKD rather than the strongest listed SSL-KD baseline DisCoR34 with R50 teacher · ImageNet vallinear probing accuracy: 62.5 -> 67.0Improved
Use MOKD rather than independent pre-trainingR50 with ViT-B · ImageNet 1% subsettop-1 accuracy: 49.5 -> 57.2Improved
Use MOKD rather than independent pre-trainingR50 with ViT-S · CIFAR-100top-1 accuracy: 85.3 -> 86.2Improved
Use MOKD rather than DisCo pre-trainingR34 with R50 teacher · COCO val2017detection APᵇ: 40.0 -> 40.3Improved
Add cross-distillation L_cm to self-distillation L_sm (ablation)MOKD, R50 and ViT-S · ImageNet100 vallinear probing top-1 accuracy: 87.0 -> 87.2 (R50); 80.3 -> 82.6 (ViT-S)Improved
Add T-Head self-distillation L_st (ablation)MOKD, R50 and ViT-S · ImageNet100 vallinear probing top-1 accuracy: 87.2 -> 87.7 (R50); 82.6 -> 83.5 (ViT-S)Improved
Add T-Head cross-distillation L_ct (ablation)MOKD, R50 and ViT-S · ImageNet100 vallinear probing top-1 accuracy: 87.7 -> 88.3 (R50); 83.5 -> 84.6 (ViT-S)Improved
Increase T-Head depth from one to three transformer blocks (ablation)MOKD, R50 and ViT-S · ImageNet100 vallinear probing top-1 accuracy: 87.4, 83.3 -> 88.3, 84.6Improved
Increase cross-distillation weight for the stronger R50 while keeping ViT-S weight at 1 (ablation)MOKD, R50-ViT-S · ImageNet100 vallinear probing top-1 accuracy: 88.3 -> 87.4 (R50); 84.6 -> 84.1 (ViT-S)Hurt
Increase cross-distillation weight for the weaker ViT-S while keeping R50 weight at 0.1 (ablation)MOKD, R50-ViT-S · ImageNet100 vallinear probing top-1 accuracy: 80.4 -> 84.6 (ViT-S)Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy