Research map / Federated learning and privacy / Communication-efficient FL
Distributed Distillation for On-Device Learning
Ilai Bistritz, Ariana Mann, Nicholas Bambos
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 90x communication reduction (tau=10, 8-bit quantization, Top-K) on top of D-Distillation | D-Distillation · MNIST / CIFAR10 | test accuracy: 46x -> 9,370x total communication reduction | No clear effect |
| reduce network batch size b communicated per round (ablation) | D-Distillation · MNIST / CIFAR10 | test accuracy | No clear effect |
| allow heterogeneous local models across devices vs D-SGD on separate subsets | D-Distillation · CIFAR10, 8 devices | test accuracy | Improved |
| increase number of devices (more private data) | D-Distillation · MNIST | test accuracy | Improved |
| D-Distillation vs Silo-SGD (no communication) | devices' classifiers · MNIST 16 devices | test accuracy | Improved |
Closest papers by meaning
- DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems (AAAI 2022 · similarity 0.86)
- Federated Learning Based on Dynamic Regularization (ICLR 2021 · similarity 0.85)
- TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge (ICML 2024 · similarity 0.84)
- A Kernel Perspective on Distillation-based Collaborative Learning (NeurIPS 2024 · similarity 0.84)
- An Empirical Study of Distributed Deep Learning Training on Edge (Student Abstract) (AAAI 2024 · similarity 0.84)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
