Research map / Label-efficient and robust learning / Knowledge distillation
TSPipe: Learn from Teacher Faster with Pipelines
Hwijoon Lim, Yechan Kim, Sukmin Yun, Jinwoo Shin, Dongsu Han
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TSPipe pipeline instead of inter-layer model parallelism | MoCo-v3 ViT-Huge, 8 GPUs · training throughput | seq/s: 5.496 -> 35.26 (6.42x) | Improved |
| use TSPipe pipeline instead of inter-layer model parallelism | BYOL ResNet-18, 8 GPUs · training throughput | seq/s: 346.3 -> 728.5 (2.10x) | Improved |
| use TSPipe instead of GPipe | MoCo-v3 ViT-Huge, 8 GPUs · training throughput | seq/s: 18.71 -> 35.26 | Improved |
| use TSPipe instead of GPipe | DistilBERT BERT-xxlarge, 8 GPUs · training throughput | seq/s: 75.22 -> 98.82 | Improved |
| scale from 8 to 16 GPUs | TSPipe on MoCo-v3 ViT-Huge · training throughput speedup over inter-layer MP | speedup: 6.42x -> 12.15x | Improved |
| scale from 8 to 16 GPUs | TSPipe on Soft Target ViT-Huge / ResNet-152 · training throughput speedup over inter-layer MP | speedup: 4.68x -> 7.21x | Improved |
| increase model size (parameters) | TSPipe · training throughput gain over MP | speedup: 2.10x -> 6.42x | Improved |
| use TSPipe instead of vanilla training | BYOL ResNet-18 · CIFAR100 linear eval | Top-1 accuracy: 48.54 -> 49.79 | Improved |
| use TSPipe instead of vanilla training | BYOL ResNet-18 · CIFAR10 linear eval | Top-1 accuracy: 74.76 -> 75.24 | No clear effect |
| use TSPipe instead of vanilla training | BYOL ResNet-18 · STL10 linear eval | Top-5 accuracy: 99.41 -> 99.40 | No clear effect |
| use TSPipe instead of vanilla training | BYOL ResNet-18 · ImageNet100 linear eval | Top-1 accuracy: 64.18 -> 64.24 | No clear effect |
| remove asymmetric parameter update (make both networks stale) (ablation) | TSPipe BYOL ResNet-18 · STL10 linear eval | Top-1 accuracy: 81.7 -> 79.1 | Hurt |
Closest papers by meaning
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.85)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.84)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.84)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.84)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.84)
- Scaling Multi-Teacher Distillation for Digital Pathology (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
