Research map / Federated learning and privacy / Personalized and heterogeneous FL
Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fed-POE, a personalized online federated learning method where each client ensembles its local model, the federated model, and a personalized subset of server-stored historical federated models, with sublinear regret guarantees.
Problem
Online prediction on streaming, non-stationary, non-i.i.d. client data: existing online federated learning gives no regret advantage over local training, and non-convex models (e.g., neural networks) suffer forgetting during online fine-tuning.
Approach
Each client adaptively combines a local online-gradient model, the global federated model, and a personalized subset of periodically saved federated checkpoints, using multiplicative-weights ensembling and importance-sampled model selection, yielding sublinear regret versus the best model in hindsight.
Key results
- On CIFAR-10, Fed-POE reaches 66.54% average client accuracy versus 65.09% for the best baseline Fed-OMD and 50.35% for Local.
- On FMNIST, Fed-POE reaches 79.23% accuracy versus 79.04% for Fed-Rep and 78.81% for Local.
- On the Air regression dataset, Fed-POE achieves MSE 9.06e-3 versus 9.12e-3 for Local; on WEC it achieves 11.83e-3 versus 16.94e-3 for POF-MKL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ensemble local and federated models with selected saved models | Fed-POE · Air | MSE (× 10^-3): 9.12 ± 3.59 -> 9.06 ± 3.73 | Helped |
| Ensemble local and federated models with selected saved models | Fed-POE · WEC | MSE (× 10^-3): 16.94 ± 15.92 -> 11.83 ± 4.60 | Helped |
| Ensemble local and federated models with selected saved models | Fed-POE · CIFAR-10 | accuracy: 65.09% ± 7.39% -> 66.54% ± 8.07% | Helped |
| Ensemble local and federated models with selected saved models | Fed-POE · FMNIST | accuracy: 79.04% ± 1.77% -> 79.23% ± 1.88% | Helped |
| Add selected saved server models (M=8 instead of M=0, b=10) (ablation) | Fed-POE · CIFAR-10 | accuracy: 65.55% ± 8.77% -> 66.54% ± 8.08% | Helped |
| Increase selected saved models from M=8 to M=16 (b=10) (ablation) | Fed-POE · CIFAR-10 | accuracy: 66.54% ± 8.08% -> 66.46% ± 7.98% | Hurt |
| Increase batch size from b=1 to b=10 (M=8) (ablation) | Fed-POE · CIFAR-10 | accuracy: 62.73% ± 8.29% -> 66.54% ± 8.08% | Helped |
| Increase batch size from b=10 to b=30 (M=8) (ablation) | Fed-POE · CIFAR-10 | accuracy: 66.54% ± 8.08% -> 66.24% ± 8.05% | Hurt |
| Increase selected saved models from M=0 to M=8 (b=1) (ablation) | Fed-POE · CIFAR-10 | accuracy: 53.80% ± 6.71% -> 62.73% ± 8.29% | Helped |
Surprising findings
- Local training outperforms several federated baselines on some datasets (e.g., Air regression, FMNIST), and the relative benefit of federated learning depends on the dataset.
- Batch size b=1 gives much worse accuracy (about 53.8% at M=0) due to forgetting, and increasing M beyond a moderate value does not further improve accuracy.
Limitations
- Regret guarantees for the non-convex neural network case are not proven; the non-convex method's guarantee relies on the model selection analysis with the convex-style ensemble bound.
- Experiments are limited to small CNNs and random-feature regression on two regression and two image datasets with 20 or 400 clients.
- Hyperparameters such as n, U, M and batch size must be chosen manually, and the paper notes fine-tuning may still forget without an appropriate batch size.
Experimental setup
Datasets: CIFAR-10 Fashion MNIST Air WEC
Benchmarks: CIFAR-10 Fashion MNIST Air WEC
Metrics: accuracy mean squared error
Baselines: Local Ditto Fed-Rep Fed-OMD Fed-ALA Fed-DS eM-KOFL POF-MKL
Builds on: FedAvg online gradient descent Ditto FedRep Fed-OMD
Closest papers by meaning
- Fine-Tuning Personalization in Federated Learning to Mitigate Adversarial Clients (NeurIPS 2024 · similarity 0.89)
- A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.87)
- pFedClub: Controllable Heterogeneous Model Aggregation for Personalized Federated Learning (NeurIPS 2024 · similarity 0.87)
- Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guarantees (ICML 2025 · similarity 0.86)
- FedAS: Bridging Inconsistency in Personalized Federated Learning (CVPR 2024 · similarity 0.86)
- Adaptive Latent-Space Constraints in Personalized Federated Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.