AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Automated Knowledge Distillation via Monte Carlo Tree Search

Lujun Li, Peijie Dong, Zimian Wei, Ya Yang

ICCV 2023 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 52 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use MCTS instead of random search for distiller search (ablation)Auto-KD · CIFAR-100 ResNet20top-1 accuracy (%): 71.55 -> 72.52Improved
use the tree-structured search space instead of a naive organization (ablation)Auto-KD · CIFAR-100 ResNet20top-1 accuracy (%): 70.32 -> 72.52 (MCTS)Improved
add extended search space operations (ablation)Auto-KD · CIFAR-100 ResNet20top-1 accuracy (%): 72.52 -> 72.65Improved
add CKA representation gap to the loss-based reward (ablation)Auto-KD · CIFAR-100 ResNet20top-1 accuracy (%): 72.36 -> 72.52Improved
use loss value as reward instead of accuracy (ablation)Auto-KD · CIFAR-100 ResNet20top-1 accuracy (%): 72.25 -> 72.36Improved
offline teacher feature storage during search (ablation)Auto-KD search · CIFAR-100 ResNet20search time / final accuracy: 6,036 s -> 4,654 s; Acc-2 72.65 -> 72.65Improved
add subsets-20% and early-stop-20% proxy settings (ablation)Auto-KD search · CIFAR-100 ResNet20final accuracy Acc-2 / time: Acc-2 72.61 -> 72.52; time 3,816 s -> 150.6 sMixed
add 50% sparse student training in search (ablation)Auto-KD search · CIFAR-100 ResNet20final accuracy Acc-2: 72.65 -> 72.61No clear effect
use train-free factors with Auto-KD for multi-layer distillationAuto-KD · CIFAR-100 multi-layertop-1 accuracy (%): 71.93 -> 72.55 (vs KR+DistPro)Improved
apply Auto-KD on object detection with distillers searched on classificationFaster RCNN-R50 / RetinaNet-R50 · MS-COCOAP: 38.40 -> 42.40; 37.40 -> 41.10Improved
Auto-KD vs MGD on RetinaNet detectionRetinaNet-R50 · MS-COCOAP: 41.00 -> 41.10No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy