Research map / Federated learning and privacy / Personalized and heterogeneous FL

Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors

ICML 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeSA, a serverless decentralized federated mutual learning method that shares distribution-matched synthetic anchor data and combines an anchor-based feature regularizer with knowledge distillation to handle both data and model heterogeneity.

Problem

Decentralized FL without a server lacks a global model, so client models generalize poorly under both data (non-iid, domain shift) and model (heterogeneous architectures) heterogeneity, and existing heterogeneous FL methods need public data or a central server.

Approach

Synthesize class-balanced anchor data per client via MMD-based distribution matching, share it among peers, and train each client with a supervised-contrastive regularizer pulling local latent features toward synthetic anchor features plus a KL distillation loss on neighbors' averaged logits over the anchors.

Key results

• On DIGITS with heterogeneous models, DeSA reaches 74.47% average global accuracy vs. 69.70% for FedProto and 67.32% for FedDF (Cifar100 public data).
• On OFFICE, DeSA achieves 54.46% average global accuracy, above FCCL's 46.07%.
• On homogeneous-model experiments, DeSA gives 95.53% average local accuracy on DIGITS vs. 94.95% for Scaffold, and 82.92% on OFFICE vs. 77.52% for Scaffold.

Measured effects reported in the paper

Surprising findings

• Naively adding synthetic anchor data without the REG and KD losses yields sub-optimal results, showing the losses are essential.
• On CIFAR10C with client sampling ratio 0.1, FCCL outperforms DeSA, which the authors attribute to pretrained-model starting points and CIFAR100 semantic overlap with CIFAR10C.

Limitations

• Synthetic data sharing raises image-level privacy concerns; DP variant reduces accuracy (DIGITS avg 74.47 to 71.12).
• Performance drops at low client sampling ratio (0.1) in CIFAR10C, where it trails FCCL and Scaffold on some metrics.
• Theoretical bound relies on near-perfect dataset distillation that is not achieved in practice.

Experimental setup

Datasets: MNIST SVHN USPS SynthDigits MNIST-M Amazon Caltech DSLR WebCam CIFAR10-C FMNIST CIFAR100

Benchmarks: DIGITS OFFICE CIFAR10C

Metrics: averaged global accuracy averaged local accuracy

Baselines: FedHe FedDF FCCL FedProto FedAvg FedProx MOON Scaffold FedGen VHL

Builds on: FedAvg VHL FedProto FedDF Dataset distillation via distribution matching

Closest papers by meaning

• Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guarantees (ICLR 2025 · similarity 0.89)
• SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning (AAAI 2025 · similarity 0.89)
• Fake It Till Make It: Federated Learning with Consensus-Oriented Generation (ICLR 2024 · similarity 0.89)
• Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.88)
• FLUX: Efficient Descriptor-Driven Clustered Federated Learning under Arbitrary Distribution Shifts (NeurIPS 2025 · similarity 0.88)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity (CVPR 2024 · similarity 0.88)