Research map / Federated learning and privacy / Communication-efficient FL
Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Local Superior Soups (LSS), a local model-interpolation training method with diversity and affinity regularization that reduces communication rounds when adapting pre-trained models in cross-silo federated learning.
Problem
Adapting large pre-trained models in federated learning under data heterogeneity requires many communication rounds, and existing model-soups approaches are too costly to run on clients.
Approach
Sequential random interpolation of a growing pool of local models during local training (no separate selection phase), regularized by a diversity term (distance among pool models) and an affinity term (distance to the pre-trained initialization), so local models stay in one connected low-loss valley and aggregate well after very few rounds.
Key results
- On FMNIST with R=1 communication round, LSS reaches 72.66% accuracy versus 63.21% for DiWA and 35.54% for FedAvg.
- On Digit-5 with R=3 rounds, LSS reaches 92.97% accuracy versus 88.83% for DiWA and 80.48% for FedAvg.
- On CIFAR-10 with R=1, LSS achieves 65.96% versus 61.32% for DiWA and 58.34% for FedAvg; the paper says LSS roughly reaches target accuracy in about 6-8 rounds where FedAvg needs hundreds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LSS instead of DiWA | pretrained federated fine-tuning · FMNIST, R=1 and R=3 | accuracy: 63.21(0.54) -> 72.66(0.73); 91.88(0.13) -> 92.45(0.21) | Helped |
| Use LSS instead of DiWA | pretrained federated fine-tuning · CIFAR-10, R=1 and R=3 | accuracy: 61.32(1.26) -> 65.96(1.50); 68.05(1.10) -> 75.16(1.07) | Helped |
| Use LSS instead of DiWA | pretrained federated fine-tuning · Digit-5, R=1 and R=3 | accuracy: 61.54(0.83) -> 72.86(1.64); 88.83(0.69) -> 92.97(0.65) | Helped |
| Use LSS instead of DiWA | pretrained federated fine-tuning · DomainNet, R=1 and R=3 | accuracy: 24.88(2.54) -> 27.86(2.85); 38.32(1.50) -> 41.35(1.46) | Helped |
| Add the affinity regularization term (ablation) | LSS on CIFAR-10 · CIFAR-10, R=1 | accuracy | Helped |
| Add the diversity regularization term (ablation) | LSS on CIFAR-10 · CIFAR-10, R=1 | accuracy | Helped |
| Tune affinity coefficient to 3 (ablation) | LSS on CIFAR-10 · CIFAR-10, R=1 | accuracy | Helped |
| Tune diversity coefficient to 3 (ablation) | LSS on CIFAR-10 · CIFAR-10, R=1 | accuracy | Helped |
| Increase the number of averaged models (ablation) | LSS · CIFAR-10 | global model accuracy | Helped |
| Increase FedAvg local steps from 8 to 12 (ablation) | FedAvg · CIFAR-10, R=1 | accuracy: 58.34(0.86) -> 55.76(0.82) | Hurt |
| Increase FedAvg local steps from 8 to 16 (ablation) | FedAvg · CIFAR-10, R=1 | accuracy: 58.34(0.86) -> 53.21(0.80) | Hurt |
| Increase averaged model count from 2 to 4 (ablation) | LSS · CIFAR-10 loss landscape | Hessian eigenvalue: 147.20 -> 119.14 | Helped |
Surprising findings
- Simply increasing local steps does not reliably help: FedAvg on CIFAR-10 drops in accuracy when local steps go beyond 8.
- LSS uses about 4x memory of other methods and trades training memory for communication savings.
Limitations
- Requires about 4x training memory relative to baselines due to multiple interpolated models.
- Evaluated only on image classification benchmarks with small client counts; language and multimodal settings left to future work.
- Theoretical analysis relies on convexity and smoothness assumptions that do not hold for deep networks.
Experimental setup
Datasets: FMNIST CIFAR-10 Digit-5 DomainNet
Benchmarks: FMNIST CIFAR-10 Digit-5 DomainNet
Metrics: test accuracy accuracy after R=1 and R=3 communication rounds sharpness (Hessian eigenvalue) noise robustness
Baselines: FedAvg FedProx MOON FedBN FedFomo FedRep FedBABU SWA SWAD Model Soups DiWA
Builds on: Model Soups DiWA FedAvg SWA
Closest papers by meaning
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.88)
- Revisiting Ensembling in One-Shot Federated Learning (NeurIPS 2024 · similarity 0.87)
- Federated Model Heterogeneous Matryoshka Representation Learning (NeurIPS 2024 · similarity 0.87)
- One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models (ICML 2025 · similarity 0.86)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.86)
- FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.