Research map / Federated learning and privacy / Communication-efficient FL
DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIMAT, a decentralized deep learning framework that periodically merges neighboring agents' models via activation-matching permutations (model merging) interleaved with local training, with convergence analysis and experiments on CIFAR-10/100 and Tiny ImageNet.
Problem
Decentralized deep learning with large pre-trained models incurs high communication and computation overhead and degrades under non-IID data and sparse topologies.
Approach
Replace vanilla weight averaging in decentralized SGD with activation-matching-based model merging (permutation-aligned merging) applied periodically between neighboring agents, which enlarges the spectral gap and speeds early accuracy gains with less communication.
Key results
- On CIFAR-100 IID with ResNet-20 and 5 agents (FC topology), DIMAT reaches 73.99±0.18% test accuracy vs 41.39±0.24% for SGP and 39.69±0.20% for CDSGD.
- On CIFAR-100 non-IID with VGG16 (FC, 5 agents), DIMAT reaches 52.99±0.24% vs 9.81±0.15% for CDSGD and 49.48±0.68% for WA.
- DIMAT achieves the best known sublinear non-convex convergence rate with a tighter error bound dependent on (1-rho') where rho' <= rho.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use activation matching for decentralized model merging instead of weight averaging | DIMAT with ResNet-20 · CIFAR-100 IID, fully connected, 5 agents | test accuracy: 73.97±0.12 -> 73.99±0.18 | No clear effect |
| Use activation matching for decentralized model merging instead of weight averaging | DIMAT with ResNet-20 · CIFAR-100 IID, ring, 5 agents | test accuracy: 58.49±0.81 -> 69.07±0.01 | Helped |
| Use activation matching for decentralized model merging instead of weight averaging | DIMAT with ResNet-20 · CIFAR-100 non-IID, fully connected, 5 agents | test accuracy: 62.7±0.44 -> 62.05±0.64 | Hurt |
| Use activation matching for decentralized model merging instead of weight averaging | DIMAT with ResNet-20 · CIFAR-100 non-IID, ring, 5 agents | test accuracy: 8.00±0.51 -> 20.45±0.65 | Helped |
| Use activation matching for decentralized model merging instead of weight averaging | DIMAT with VGG16 · CIFAR-100 IID, fully connected, 5 agents | test accuracy: 66.70±0.34 -> 68.08±0.24 | Helped |
| Use activation matching for decentralized model merging instead of weight averaging | DIMAT with VGG16 · CIFAR-100 IID, ring, 5 agents | test accuracy: 67.02±0.24 -> 63.76±0.30 | Hurt |
| Use activation matching for decentralized model merging instead of weight averaging | DIMAT with VGG16 · CIFAR-100 non-IID, fully connected, 5 agents | test accuracy: 49.48±0.68 -> 52.99±0.24 | Helped |
| Use activation matching for decentralized model merging instead of weight averaging | DIMAT with VGG16 · CIFAR-100 non-IID, ring, 5 agents | test accuracy: 25.15±0.27 -> 25.28±0.13 | No clear effect |
Surprising findings
- CGA failed to converge in the explored setting and was omitted from experiments.
- Weight Averaging eventually matches DIMAT's accuracy given enough training time, so DIMAT's advantage is mainly faster early gains.
Limitations
- CGA could not be run/converge in the explored setting, so it was omitted from comparisons.
- Scalability issues under non-IID data with more agents; accuracy per agent diminishes as agent count grows.
- Weight Matching is not implemented for ResNet, so comparisons are limited for ResNet architectures.
Experimental setup
Datasets: CIFAR-100 CIFAR-10 Tiny ImageNet
Benchmarks: CIFAR-100 CIFAR-10 Tiny ImageNet
Metrics: test accuracy convergence rate
Baselines: SGP CDSGD CGA Weight Matching Weight Averaging
Builds on: activation matching (Ainsworth et al.) D-PSGD CDSGD Stochastic Gradient Push
Closest papers by meaning
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.88)
- Consensus Control for Decentralized Deep Learning (ICML 2021 · similarity 0.88)
- Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings (NeurIPS 2025 · similarity 0.85)
- ZipIt! Merging Models from Different Tasks without Training (ICLR 2024 · similarity 0.84)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.83)
- Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.