AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

TSPipe: Learn from Teacher Faster with Pipelines

Hwijoon Lim, Yechan Kim, Sukmin Yun, Jinwoo Shin, Dongsu Han

ICML 2022 · Label-efficient and robust learning › Knowledge distillation · approach: Teacher properties and robustness

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use TSPipe pipeline instead of inter-layer model parallelismMoCo-v3 ViT-Huge, 8 GPUs · training throughputseq/s: 5.496 -> 35.26 (6.42x)Improved
use TSPipe pipeline instead of inter-layer model parallelismBYOL ResNet-18, 8 GPUs · training throughputseq/s: 346.3 -> 728.5 (2.10x)Improved
use TSPipe instead of GPipeMoCo-v3 ViT-Huge, 8 GPUs · training throughputseq/s: 18.71 -> 35.26Improved
use TSPipe instead of GPipeDistilBERT BERT-xxlarge, 8 GPUs · training throughputseq/s: 75.22 -> 98.82Improved
scale from 8 to 16 GPUsTSPipe on MoCo-v3 ViT-Huge · training throughput speedup over inter-layer MPspeedup: 6.42x -> 12.15xImproved
scale from 8 to 16 GPUsTSPipe on Soft Target ViT-Huge / ResNet-152 · training throughput speedup over inter-layer MPspeedup: 4.68x -> 7.21xImproved
increase model size (parameters)TSPipe · training throughput gain over MPspeedup: 2.10x -> 6.42xImproved
use TSPipe instead of vanilla trainingBYOL ResNet-18 · CIFAR100 linear evalTop-1 accuracy: 48.54 -> 49.79Improved
use TSPipe instead of vanilla trainingBYOL ResNet-18 · CIFAR10 linear evalTop-1 accuracy: 74.76 -> 75.24No clear effect
use TSPipe instead of vanilla trainingBYOL ResNet-18 · STL10 linear evalTop-5 accuracy: 99.41 -> 99.40No clear effect
use TSPipe instead of vanilla trainingBYOL ResNet-18 · ImageNet100 linear evalTop-1 accuracy: 64.18 -> 64.24No clear effect
remove asymmetric parameter update (make both networks stale) (ablation)TSPipe BYOL ResNet-18 · STL10 linear evalTop-1 accuracy: 81.7 -> 79.1Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy