Research map / Federated learning and privacy / Communication-efficient FL
NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NTK-DFL, a decentralized federated learning method that evolves client weights via a local NTK built from exchanged Jacobians, combined with per-round neighbor averaging and final model averaging, to handle data heterogeneity.
Problem
Statistical heterogeneity degrades decentralized federated learning (DFL) without a central server; existing DFL methods converge slowly and underperform under non-IID data.
Approach
Clients exchange Jacobians of neighbors' models on their local data, build a local neural tangent kernel over their neighborhood, and evolve weights in closed form via NTK dynamics; per-round neighbor weight averaging and a final validation-ranked model averaging exploit inter-client deviation.
Key results
- On Fashion-MNIST with alpha=0.1, NTK-DFL reaches 85% test accuracy in 4.6 times fewer communication rounds than DFedAvg, the next best baseline.
- Under alpha=0.1 NTK-DFL establishes a 3-4% accuracy lead over the best baseline within five communication rounds.
- Final model averaging yields an aggregated model about 10% more accurate than the mean client accuracy in the most heterogeneous setting (about 15% with kappa=2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NTK-based decentralized weight evolution with model averaging instead of the best baseline | DFL on Fashion-MNIST, α=0.1 · Fashion-MNIST α=0.1 | communication rounds to 85% test accuracy: 4.6 times fewer communication rounds | Helped |
| use NTK-DFL rather than the best-performing baseline | DFL on Fashion-MNIST · Fashion-MNIST α=0.1 | test accuracy: 3--4% accuracy lead | Helped |
| use NTK-DFL rather than baselines across sparsity levels | DFL on Fashion-MNIST · Fashion-MNIST, varying sparsity | test accuracy: 2--3% | Helped |
| add per-round neighbor parameter averaging (ablation) | NTK-DFL · Fashion-MNIST | client-model accuracy distribution | Helped |
| add final model averaging across clients (ablation) | NTK-DFL · Fashion-MNIST α=0.1 | test accuracy: nearly 10% | Helped |
| increase the number of Jacobian batches (ablation) | NTK-DFL · Fashion-MNIST α=0.1 | test accuracy | Helped |
| subsample client data each communication round (ablation) | NTK-DFL · Fashion-MNIST α=0.1 | test accuracy | Hurt |
| use a dynamic rather than static network topology (ablation) | NTK-DFL · Fashion-MNIST α=0.1 | convergence | Helped |
| use random per-client initialization rather than uniform initialization (ablation) | NTK-DFL · Fashion-MNIST α=0.1 | convergence | Hurt |
| use a ring or line topology rather than a topology with more neighbors (ablation) | NTK-DFL · Fashion-MNIST | convergence | Hurt |
| increase the network density by decreasing sparsity (ablation) | NTK-DFL · Fashion-MNIST | convergence accuracy | Helped |
| increase data heterogeneity by decreasing α (ablation) | NTK-DFL · Fashion-MNIST | test accuracy | No clear effect |
Surprising findings
- Jacobian batching (more batches) counterintuitively improves test accuracy per round.
- Baseline gradient-based methods could not withstand the aggressive sparsification applied to NTK-DFL, so only quantization was used for baselines.
Limitations
- Per-round communication volume is higher than weight-based methods since Jacobians are shared; compression needed to reduce it.
- NTK approximation error delta_NTK is treated as a constant in the theory, leaving an irreducible convergence floor.
- Experiments limited to small MLPs on MNIST-style image datasets; extension to CNNs and transformers left to future work.
Experimental setup
Datasets: Fashion-MNIST FEMNIST MNIST
Benchmarks: Fashion-MNIST FEMNIST MNIST
Metrics: test accuracy communication rounds to 85% accuracy convergence rate
Baselines: D-PSGD DFedAvg DFedAvgM DFedSAM DisPFL NTK-FL
Builds on: FedAvg NTK-FL D-PSGD DFedAvg DisPFL
Closest papers by meaning
- Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guarantees (ICLR 2025 · similarity 0.91)
- SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning (AAAI 2025 · similarity 0.91)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.88)
- One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models (ICML 2025 · similarity 0.88)
- Decentralized Directed Collaboration for Personalized Federated Learning (CVPR 2024 · similarity 0.88)
- Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.