AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Knowledge Flow: Improve Upon Your Teachers

Iou-Jen Liu, Jian Peng, Alexander Schwing

ICLR 2019 · Label-efficient and robust learning › Knowledge distillation · approach: Teacher properties and robustness · 17 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use knowledge flow with CIFAR-100 and SVHN teachers instead of the baselineDenseNet · CIFAR-10 testtop-1 error (%): 4.44 -> 3.88Improved
use knowledge flow with CIFAR-10 and SVHN teachers instead of the baselineDenseNet · CIFAR-100 testtop-1 error (%): 21.64 -> 20.78Improved
combine two teachers rather than use oneKnowledge Flow Atari student · Atari gamesscoreImproved
use knowledge flow with non-expert teachers instead of fine-tuning from one non-expertA3C student · Alien, Breakout, ChopperCommand, KungFuMaster, MsPacman, SeaquestscoreImproved
use untrained teachers instead of knowledgeable teachers (ablation)Knowledge Flow A3C student · Heroaverage reward: 30928 -> 15934Hurt
remove the KL regularizer during teacher influence reduction (ablation)Knowledge Flow A3C student · MsPacmanaverage reward: 2907 -> 1215Hurt
use teachers with different architectures from the student (ablation)Knowledge Flow A3C student · KungFuMasteraverage reward: 35012 -> 37520No clear effect
compute the previous model with an exponential running average rather than a single model (ablation)Knowledge Flow A3C student · Boxingaverage reward: 96.0 -> 96.2No clear effect
link a teacher's bottom layer to a student's top layer (ablation)Knowledge Flow student · General experimentsperformance improvementNo clear effect
use knowledge flow rather than knowledge distillationSmaller supervised student models · MNIST, MNIST without digit 3, CIFAR-100, ImageNettest error (%): 0.74 -> 0.73; 2.06 -> 1.05; 30.28 -> 30.07; 30.04 -> 29.05Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy