Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Flow: Improve Upon Your Teachers
Iou-Jen Liu, Jian Peng, Alexander Schwing
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use knowledge flow with CIFAR-100 and SVHN teachers instead of the baseline | DenseNet · CIFAR-10 test | top-1 error (%): 4.44 -> 3.88 | Improved |
| use knowledge flow with CIFAR-10 and SVHN teachers instead of the baseline | DenseNet · CIFAR-100 test | top-1 error (%): 21.64 -> 20.78 | Improved |
| combine two teachers rather than use one | Knowledge Flow Atari student · Atari games | score | Improved |
| use knowledge flow with non-expert teachers instead of fine-tuning from one non-expert | A3C student · Alien, Breakout, ChopperCommand, KungFuMaster, MsPacman, Seaquest | score | Improved |
| use untrained teachers instead of knowledgeable teachers (ablation) | Knowledge Flow A3C student · Hero | average reward: 30928 -> 15934 | Hurt |
| remove the KL regularizer during teacher influence reduction (ablation) | Knowledge Flow A3C student · MsPacman | average reward: 2907 -> 1215 | Hurt |
| use teachers with different architectures from the student (ablation) | Knowledge Flow A3C student · KungFuMaster | average reward: 35012 -> 37520 | No clear effect |
| compute the previous model with an exponential running average rather than a single model (ablation) | Knowledge Flow A3C student · Boxing | average reward: 96.0 -> 96.2 | No clear effect |
| link a teacher's bottom layer to a student's top layer (ablation) | Knowledge Flow student · General experiments | performance improvement | No clear effect |
| use knowledge flow rather than knowledge distillation | Smaller supervised student models · MNIST, MNIST without digit 3, CIFAR-100, ImageNet | test error (%): 0.74 -> 0.73; 2.06 -> 1.05; 30.28 -> 30.07; 30.04 -> 29.05 | Improved |
Closest papers by meaning
- Improved Knowledge Distillation via Teacher Assistant (AAAI 2020 · similarity 0.90)
- Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (ICCV 2019 · similarity 0.89)
- Densely Guided Knowledge Distillation Using Multiple Teacher Assistants (ICCV 2021 · similarity 0.88)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.87)
- Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · similarity 0.87)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
