AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Weight Decay Scheduling and Knowledge Distillation for Active Learning

Juseung Yun, Byungjoo Kim, Junmo Kim

ECCV 2020 · Label-efficient and robust learning › Knowledge distillation · approach: Few-shot and metric distillation · 12 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add random re-initialization each round to weight decay scheduling (WS+init vs WS) (ablation)WS (weight decay scheduling) · CIFAR-10 / CIFAR-100accuracyImproved
add weight decay scheduling to Entropy+init at equal final weight decay (ablation)Entropy+init, Resnet18 · CIFAR-10 (K=1000)accuracy (%): 91.51 -> 91.68Improved
add weight decay scheduling to Entropy+init at equal final weight decay (ablation)Entropy+init, Densenet100 · CIFAR-10 (K=1000)accuracy (%): 90.43 -> 89.96Hurt
add weight decay scheduling to Entropy+init on CIFAR-100 (ablation)Entropy+init, Resnet18 · CIFAR-100 (K=1000)accuracy (%): 57.43 -> 57.38No clear effect
add weight decay scheduling to Entropy+init on CIFAR-100 (ablation)Entropy+init, Densenet100 · CIFAR-100 (K=1000)accuracy (%): 55.23 -> 54.88Hurt
add knowledge distillation from previous-round teacher (KD2) to WS+init (ablation)WS+init · MNIST (2,500 labels)accuracy (%): 99.17 -> 99.29Improved
add KD1 with alpha 0.9 to WS+init (ablation)WS+init, Resnet18 · CIFAR-10 (10000 labels)accuracy (%): 91.68 -> 92.18Improved
add KD1 with alpha 0.5 at 1000 labels (large teacher-student gap) (ablation)WS+init, Resnet18 · CIFAR-10 (1000 labels)accuracy (%): 62.17 -> 61.27Hurt
use small alpha 0.1 for distillation (ablation)WS+init+KD1 · CIFAR-10 (10000 labels)accuracy (%): 91.68 -> 91.90No clear effect
keep a strong fixed weight decay 0.02 as the data grows (ablation)Resnet18 fixed WD · CIFAR-10 (50k labels)accuracy (%): 95.26 -> 10.00Hurt
keep a strong fixed weight decay 0.2 as the data grows (ablation)5-layer CNN fixed WD · MNIST (10k labels)accuracy (%): 98.98 -> 94.50Hurt
use Random instead of Entropy acquisition in early rounds with very few labelsDensenet100 · CIFAR-10 (K=250, third round)accuracy (%): 49.31 -> 51.37Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy