Research map / Architectures and efficiency / RNNs and normalization
Incorporating Serverless Computing into P2P Networks for ML Training: In-Database Tasks and Their Scalability Implications (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a P2P distributed ML training architecture that offloads gradient computation to serverless functions and runs gradient averaging and model updates inside an enhanced Redis/RedisAI in-memory database.
Problem
P2P distributed ML training suffers from resource consumption and communication overhead as peer count grows, and stateless serverless functions need strong state/communication support.
Approach
Combine serverless computing with P2P training: peers offload gradient computation to parallel serverless functions, store state in a per-peer Redis database, and execute gradient averaging and model parameter updates in-database using a tailored RedisAI.
Key results
- Serverless execution reduced computation time for VGG11 on MNIST at batch size 64 from 394.8 s to 10.5 s, at a higher cost ($0.05435 vs $0.01017 per peer).
- In-database gradient averaging with batch size 1024 took 37.41 s versus 67.32 s outside the database (44% reduction).
- In-database model updates took a consistent 4.8 s versus about 27.5 s outside the database (82% reduction).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Offload computation to serverless parallel processing | P2P training with VGG11 · MNIST, batch size 64 / 235 batches | time (sec): 394.8 -> 10.5 | Helped |
| Offload computation to serverless parallel processing | P2P training with VGG11 · MNIST, batch size 64 / 235 batches | cost (USD): 0.01017 -> 0.05435 | Hurt |
| Execute gradient averaging within the database | P2P training with ResNet-18 · batch size 1024 / 15 batches | gradient averaging time (sec): 67,32 -> 37,41 | Helped |
| Execute model updates within the database | P2P training with ResNet-18 · batch size 1024 / 15 batches | model update time (sec): 27,52 -> 4,8 | Helped |
| Increase batch size from 64 to 1024 | Serverless P2P training with DenseNet-121 · MNIST, intra-peer scalability | gradient computation time | Hurt |
| Increase batch size, reducing the number of gradients | Serverless P2P training with DenseNet-121 · MNIST, gradient averaging | gradient averaging time | Helped |
| Add more peers | Serverless P2P training with DenseNet-121 · MNIST, larger batches | computation time | No clear effect |
Surprising findings
- Adding more peers does not significantly decrease computation time for larger batches because the gains are offset by network communication overhead.
- Serverless execution is much faster but costs more per peer than the single-machine baseline.
Limitations
- Preliminary results on a single small dataset (MNIST) with only three models
- Serverless approach incurs higher monetary cost than a single machine
- Communication overhead grows with peer count, limiting inter-peer scaling
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: gradient computation time (s) gradient averaging time (s) model update time (s) total time (s) cost (USD)
Baselines: Training on a single t2.large machine without serverless Gradient averaging and model update outside the database
Builds on: RedisAI
Closest papers by meaning
- The Promise of Serverless Computing within Peer-to-Peer Architectures for Distributed ML Training (AAAI 2024 · similarity 0.95)
- Coded Computing for Resilient Distributed Computing: A Learning-Theoretic Framework (NeurIPS 2024 · similarity 0.84)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.82)
- Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML (NeurIPS 2024 · similarity 0.82)
- Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing (AAAI 2024 · similarity 0.82)
- CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.