Research map / Optimization / Distributed and compressed
DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DePRL, a fully decentralized personalized learning algorithm that shares a global representation via consensus while keeping local heads private, and proves O(1/sqrt(NK)) convergence (linear speedup) for non-linear representations.
Problem
Data heterogeneity in decentralized learning makes a single shared model perform poorly on individual workers; existing personalized decentralized methods lack convergence guarantees with shared non-linear representations.
Approach
Each worker keeps a personalized local head and a consensus-averaged shared global representation; workers alternate tau SGD steps on heads, one SGD step on the representation, and a neighbor-averaging consensus step on the representation only. The analysis uses a new epsilon-approximation notion coupling the two parameters and proves linear speedup in the number of workers.
Key results
- DePRL outperforms D-PSGD and DisPFL across all datasets, Dirichlet heterogeneity levels, and Ring/Random/FC graphs; e.g., CIFAR-100 (ResNet-18), pi=0.1, Ring: DePRL 60.72 vs DisPFL 46.09 vs D-PSGD 25.18.
- Theory gives convergence rate O(1/sqrt(NK)) with dominant terms, i.e., linear speedup in the number of workers.
- On 64 unseen workers, DePRL reaches e.g. 53.58 on CIFAR-100 (Ring, pi=0.1) vs 38.64 for D-PSGD and 34.67 for DisPFL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use shared global representations with personalized local heads instead of a personalized single shared model | Decentralized learning · CIFAR-100, Random graph, π=0.1/0.3/0.5 | test accuracy: 47.77 -> 61.51; 39.91 -> 50.49; 37.39 -> 46.68 | Helped |
| Use shared global representations with personalized local heads instead of a personalized single shared model | Decentralized learning · CIFAR-10, Random graph, π=0.1/0.3/0.5 | test accuracy: 89.01 -> 91.03; 77.36 -> 79.60; 73.30 -> 74.80 | Helped |
| Use shared global representations with personalized local heads instead of a personalized single shared model | Decentralized learning · Fashion-MNIST, Random graph, π=0.1/0.3/0.5 | test accuracy: 96.59 -> 97.16; 93.47 -> 94.81; 91.99 -> 93.55 | Helped |
| Use shared global representations with personalized local heads instead of a personalized single shared model | Decentralized learning · HARBox, Random graph, π=0.1/0.3/0.5 | test accuracy: 91.36 -> 92.49; 80.35 -> 81.30; 74.72 -> 76.22 | Helped |
| Increase the number of workers | DePRL · Convergence time | speedup | Helped |
Surprising findings
- DisPFL, a personalized method, sometimes underperforms the non-personalized D-PSGD on generalization to new workers (e.g., CIFAR-100 Ring pi=0.1: 34.67 vs 38.64).
Limitations
- Analysis assumes a one-step representation update per round and bounded variance/heterogeneity assumptions; results are for SGD without momentum or multi-step representation updates.
- Experiments use only image and HAR datasets with 128 simulated workers, and no real decentralized deployment is reported.
- Communication compression and privacy are left to future work.
Experimental setup
Datasets: CIFAR-100 CIFAR-10 Fashion-MNIST HARBox
Benchmarks: CIFAR-100 CIFAR-10 Fashion-MNIST HARBox
Metrics: average per-worker test accuracy convergence speedup generalization accuracy to new workers
Baselines: D-PSGD DisPFL FedAvg FedRep Ditto FedRoD
Builds on: D-PSGD DisPFL shared representation learning for personalized FL (Collins et al.) two-timescale stochastic approximation
Closest papers by meaning
- Decentralized Directed Collaboration for Personalized Federated Learning (CVPR 2024 · similarity 0.88)
- SP-CACW: Convergence-Aware Client Weighting for Selfish Personalized Learning (NeurIPS 2026 · similarity 0.86)
- PDUDT: Provable Decentralized Unlearning under Dynamic Topologies (ICML 2025 · similarity 0.86)
- Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors (ICML 2024 · similarity 0.85)
- Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guarantees (ICLR 2025 · similarity 0.85)
- Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.