Research map / Label-efficient and robust learning / Knowledge distillation
Weight Decay Scheduling and Knowledge Distillation for Active Learning
Juseung Yun, Byungjoo Kim, Junmo Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add random re-initialization each round to weight decay scheduling (WS+init vs WS) (ablation) | WS (weight decay scheduling) · CIFAR-10 / CIFAR-100 | accuracy | Improved |
| add weight decay scheduling to Entropy+init at equal final weight decay (ablation) | Entropy+init, Resnet18 · CIFAR-10 (K=1000) | accuracy (%): 91.51 -> 91.68 | Improved |
| add weight decay scheduling to Entropy+init at equal final weight decay (ablation) | Entropy+init, Densenet100 · CIFAR-10 (K=1000) | accuracy (%): 90.43 -> 89.96 | Hurt |
| add weight decay scheduling to Entropy+init on CIFAR-100 (ablation) | Entropy+init, Resnet18 · CIFAR-100 (K=1000) | accuracy (%): 57.43 -> 57.38 | No clear effect |
| add weight decay scheduling to Entropy+init on CIFAR-100 (ablation) | Entropy+init, Densenet100 · CIFAR-100 (K=1000) | accuracy (%): 55.23 -> 54.88 | Hurt |
| add knowledge distillation from previous-round teacher (KD2) to WS+init (ablation) | WS+init · MNIST (2,500 labels) | accuracy (%): 99.17 -> 99.29 | Improved |
| add KD1 with alpha 0.9 to WS+init (ablation) | WS+init, Resnet18 · CIFAR-10 (10000 labels) | accuracy (%): 91.68 -> 92.18 | Improved |
| add KD1 with alpha 0.5 at 1000 labels (large teacher-student gap) (ablation) | WS+init, Resnet18 · CIFAR-10 (1000 labels) | accuracy (%): 62.17 -> 61.27 | Hurt |
| use small alpha 0.1 for distillation (ablation) | WS+init+KD1 · CIFAR-10 (10000 labels) | accuracy (%): 91.68 -> 91.90 | No clear effect |
| keep a strong fixed weight decay 0.02 as the data grows (ablation) | Resnet18 fixed WD · CIFAR-10 (50k labels) | accuracy (%): 95.26 -> 10.00 | Hurt |
| keep a strong fixed weight decay 0.2 as the data grows (ablation) | 5-layer CNN fixed WD · MNIST (10k labels) | accuracy (%): 98.98 -> 94.50 | Hurt |
| use Random instead of Entropy acquisition in early rounds with very few labels | Densenet100 · CIFAR-10 (K=250, third round) | accuracy (%): 49.31 -> 51.37 | Improved |
Closest papers by meaning
- Active Data Curation Effectively Distills Large-Scale Multimodal Models (CVPR 2025 · similarity 0.87)
- Active Learning Guided by Efficient Surrogate Learners (AAAI 2024 · similarity 0.86)
- A Cross-Domain Benchmark for Active Learning (NeurIPS 2024 · similarity 0.86)
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.86)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.86)
- Active Prompt Learning in Vision Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
