Research map / Federated learning and privacy / Personalized and heterogeneous FL
PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PerAda, a parameter-efficient personalized FL framework that trains small adapters on a frozen pretrained model, regularizes personalized adapters toward a global adapter, and forms the global adapter via server-side ensemble knowledge distillation, with generalization and convergence guarantees.
Problem
Personalized federated learning must balance low communication/computation cost against personalization that generalizes to test-time distribution shifts; full-model pFL is costly and partial-model pFL overfits to local data.
Approach
Personalized adapters on a frozen pretrained ResNet are regularized toward a global adapter, which is aggregated from client local adapters by server-side ensemble knowledge distillation on unlabeled public data, so only adapters are trained and communicated (about 12.6% of parameters).
Key results
- PerAda achieves +4.85% personalized performance on CheXpert over baselines while training and communicating only about 12.6% (1.41M) of ResNet-18 parameters.
- PerAda improves out-of-distribution generalization by +5.23% on CIFAR-10 (Global-test) and +4.21% on CIFAR-10-C relative to the best baseline.
- Under DP with ViT-S/16 on CIFAR-10, PerAda retains high personalized accuracy (about 98% at epsilon=1 on local test) and improves utility over DITTO at matched privacy budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add server-side knowledge distillation to the global adapter (ablation) | PerAda · CIFAR-10 Global-test | accuracy: 53.81± 6.27 -> 59.05± 5.24 | Helped |
| add server-side knowledge distillation to the global adapter (ablation) | PerAda · Office-Home Global-test | accuracy: 76.55± 2.47 -> 77.2 ± 1.63 | Helped |
| add server-side knowledge distillation to the global adapter (ablation) | PerAda · CheXpert Global-test | AUC: 77.59± 2.18 -> 77.88± 1.55 | Helped |
| add server-side knowledge distillation to the global adapter (ablation) | PerAda · CIFAR-10 Local-test | accuracy: 91.27± 5.15 -> 91.82± 4.43 | Helped |
| increase the number of server distillation steps (ablation) | PerAda · CIFAR-10 | generalization accuracy | Helped |
| increase the number of distillation data samples (ablation) | PerAda · CIFAR-10 | generalization accuracy | Helped |
| use out-of-domain STL-10 or CIFAR-100 distillation data instead of no distillation data (ablation) | PerAda · CIFAR-10 | generalization accuracy | Helped |
| use in-domain CIFAR-10 validation data for distillation instead of no distillation data (ablation) | PerAda · CIFAR-10 | generalization accuracy | Helped |
| moderately increase the personalization regularization strength λ (ablation) | PerAda · CIFAR-10 | generalization accuracy | Helped |
| overly increase the personalization regularization strength λ (ablation) | PerAda · CIFAR-10 | personalized accuracy | Hurt |
| initialize from an ImageNet-pretrained model instead of a FEDAVG-trained model (ablation) | PerAda · CIFAR-10 | generalization accuracy | Helped |
| train with Dir(1) rather than Dir(0.3) client data heterogeneity (ablation) | PerAda · CheXpert Local-test | AUC: 76.98 ± 1.81 -> 77.47 ± 1.54 | No clear effect |
Surprising findings
- Existing partial-model pFL methods with adapters (FedALT, FedSim) generalize poorly to test-time shifts despite competitive personalized accuracy.
- Updating shared parameters globally via FL can compromise the pretrained feature extractor; keeping the backbone frozen (PerAda w/o KD) generalizes better.
Limitations
- Convergence and generalization analysis assumes bounded gradients and covariate-shift non-IID settings; the bounds depend on the distillation distribution's closeness to the target.
- Requires an unlabeled public distillation dataset on the server, whose choice affects generalization.
Experimental setup
Datasets: CIFAR-10 Office-Home CheXpert CIFAR-100 STL-10 CIFAR-10.1 CIFAR-10-C
Benchmarks: CIFAR-10 Office-Home CheXpert CIFAR-10.1 CIFAR-10-C
Metrics: test accuracy (pFL accuracy) AUC number of trainable parameters number of communicated parameters epsilon under differential privacy
Baselines: FedAvg+FT Ditto APFL MTL pFedMe FedBN LG-FedAvg FedRep FedALT FedSim FedDF FedProx
Builds on: FedAvg Adapter FedDF (ensemble distillation) Ditto FedALT FedSim
Closest papers by meaning
- Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guarantees (ICML 2025 · similarity 0.90)
- FedAS: Bridging Inconsistency in Personalized Federated Learning (CVPR 2024 · similarity 0.89)
- FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning (CVPR 2024 · similarity 0.89)
- Flexible Sharpness-Aware Personalized Federated Learning (AAAI 2025 · similarity 0.88)
- Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning (AAAI 2025 · similarity 0.88)
- PerFedRLNAS: One-for-All Personalized Federated Neural Architecture Search (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.