Research map / Architectures and efficiency / RNNs and normalization
An Empirical Study of Distributed Deep Learning Training on Edge (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of distributed ResNet-18 training on CIFAR-10 across a cluster of Raspberry Pi 4 edge devices, comparing Chainer, PyTorch and TensorFlow on throughput, memory, scaling and hardware counters.
Problem
Deep learning training is expensive and inaccessible without supercomputers or high-end GPUs; it is unclear how well low-energy, low-cost edge devices with ARM CPUs can train DL models in a distributed fashion.
Approach
Builds small Raspberry Pi 4 clusters and measures distributed training of ResNet-18 on CIFAR-10 across three DL frameworks, combining software-level metrics with hardware performance counters to expose framework inefficiencies on low-end CPUs.
Key results
- Chainer finishes training in about 47% and 53% of the time of PyTorch and TensorFlow respectively at batch size 256.
- TensorFlow is limited to batch size 512 and PyTorch to 1024 due to memory consumption; TensorFlow uses the most memory and PyTorch the least.
- A single Raspberry Pi 4 reaches 15.5 GFLOPS on LINPACK and a 4-node cluster reaches 47 GFLOPS; multi-node scaling is observed but not linear.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Chainer instead of PyTorch | ResNet-18 training on Raspberry Pi 4 · CIFAR-10, batch size 256 | training time: ≈47% of the time taken by PyTorch | Helped |
| Use Chainer instead of TensorFlow | ResNet-18 training on Raspberry Pi 4 · CIFAR-10, batch size 256 | training time: 53% of the time taken by TensorFlow | Helped |
| Scale from one Raspberry Pi 4 to a 4-node cluster | LINPACK · Raspberry Pi 4 cluster | GFLOPS: 15.5 GFLOPS; 47 GFLOPS | Helped |
| Use TensorFlow rather than other evaluated frameworks | ResNet-18 training on Raspberry Pi 4 · CIFAR-10 | L1 cache miss ratio: ≈16% | Hurt |
Surprising findings
- Chainer was the easiest framework to install for single- and multi-node setups and was fastest in the tested settings.
- PyTorch has the most instructions and memory accesses but also the highest instruction throughput.
Limitations
- Scaling across nodes is non-linear and communication bottlenecks remain.
- Evaluation limited to a small Raspberry Pi 4 cluster (up to 4 nodes) and a single model and dataset.
- Initial results only; expanded evaluation sets are future work.
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10 LINPACK
Metrics: training throughput (images/s) epoch duration (s) USS memory (bytes) GFLOPS instructions executed L1 cache miss ratio branch mispredict ratio scaling across nodes
Baselines: PyTorch TensorFlow Chainer
Builds on: LINPACK
Closest papers by meaning
- Integrated Hardware Architecture and Device Placement Search (ICML 2024 · similarity 0.85)
- Stepping Forward on the Last Mile (NeurIPS 2024 · similarity 0.85)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.84)
- TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge (ICML 2024 · similarity 0.84)
- Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters (AAAI 2025 · similarity 0.83)
- Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.