Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedSOL, a federated learning method that perturbs local weights along the proximal (KL-divergence) gradient with an adaptive radius and then updates with the local gradient computed at the perturbed weights, so local updates stay near-orthogonal to the global-knowledge-preserving proximal direction.
Problem
Data heterogeneity in federated learning causes local training to forget global knowledge, and proximal restrictions that preserve it also hinder local learning by conflicting with the local objective.
Approach
Instead of projecting out the proximal gradient (which hurts performance), use the proximal gradient only to adversarially perturb weights (SAM-style) and compute the local gradient at the perturbed point, implicitly finding a parameter region where the local update is orthogonal to the proximal gradient; adds layer-wise adaptive perturbation radius and partial (head-only) perturbation.
Key results
- On CIFAR-10 with sharding s=2, FedSOL reaches 66.72% accuracy vs 51.48% for FedAvg.
- On CIFAR-10 LDA alpha=0.05, FedSOL reaches 60.01% vs 42.27% for FedAvg, while FedProx and Scaffold fail to exceed it in several settings.
- FedSOL achieves the best results in most settings across six datasets and three architectures (VGG-11, ResNet-18, SL-ViT), with gains over FedAvg in all reported scenarios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedSOL instead of FedNTD | federated learning · CIFAR-10, LDA α=0.05 | test accuracy@1 (%): 58.08(0.48) -> 60.01(0.30) | Helped |
| Use FedSOL instead of FedNTD | federated learning · SVHN, Sharding s=2 | test accuracy@1 (%): 85.30(0.78) -> 85.18(0.37) | No clear effect |
| Use FedSOL instead of FedNTD | federated learning · CIFAR-10, Sharding s=2 | test accuracy@1 (%): 67.25(1.08) -> 66.72(0.61) | Hurt |
| Replace direct proximal-gradient projection with FedSOL proximal perturbation (ablation) | CIFAR-10, LDA α=0.1 · CIFAR-10, LDA α=0.1 | test accuracy: 56.35±2.85 -> 60.31±2.07 | Helped |
| Add proximal-gradient projection rather than using the KL proximal loss as an auxiliary objective (ablation) | CIFAR-10, LDA α=0.1 · CIFAR-10, LDA α=0.1 | test accuracy: 60.31±2.07 -> 50.88±3.55 | Hurt |
| Add adaptive perturbation strength based on layer-wise global-local parameter discrepancy (ablation) | FedSOL · CIFAR-10, LDA α=0.1 | test accuracy | Helped |
| Perturb only the classifier head instead of the full model (ablation) | FedSOL · CIFAR-10, LDA α=0.1 | test accuracy: 64.16 -> 64.13 | No clear effect |
| Perturb all layers except the classifier head instead of the full model (ablation) | FedSOL · CIFAR-10, LDA α=0.1 | test accuracy: 64.16 -> 63.94 | Hurt |
| Combine FedSOL perturbation with FedProx (ablation) | FedProx · CIFAR-10, LDA α=0.1 | test accuracy: 59.80 -> 63.93 | Helped |
| Combine FedSOL perturbation with FedNova (ablation) | FedNova · SVHN, LDA α=0.1 | test accuracy: 53.07 -> 79.95 | Helped |
| Increase FedSOL perturbation strength ρ (ablation) | FedSOL · CIFAR-10, LDA α=0.1 | test accuracy | Mixed |
Surprising findings
- Directly projecting out the proximal gradient (gradient projection) degrades performance, even below FedAvg (e.g., KL-divergence base 60.31 vs projection 50.88 on CIFAR-10 LDA alpha=0.1).
- Many recent FL methods (FedNova, Scaffold, FedSAM in some settings) underperform the FedAvg baseline.
Limitations
- Proximal loss must be computed additionally, increasing local compute (FLOPs 1.33x-2x+delta).
- Theoretical analysis relies on first-order Taylor approximations and a PSD Hessian assumption.
Experimental setup
Datasets: MNIST CIFAR-10 SVHN CINIC-10 PathMNIST TissueMNIST
Benchmarks: MNIST CIFAR-10 SVHN CINIC-10 PathMNIST TissueMNIST
Metrics: test accuracy@1 weight divergence loss landscape top eigenvalue ratio
Baselines: FedAvg FedProx FedNova Scaffold FedNTD FedSAM FedASAM FedDyn MOON
Builds on: FedAvg SAM FedProx FedNTD orthogonal gradient projection in continual learning
Closest papers by meaning
- Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning (ICLR 2024 · similarity 0.88)
- FedImpro: Measuring and Improving Client Update in Federated Learning (ICLR 2024 · similarity 0.87)
- FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.86)
- Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization (ICML 2024 · similarity 0.86)
- Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable (NeurIPS 2025 · similarity 0.86)
- pFedMxF: Personalized Federated Class-Incremental Learning with Mixture of Frequency Aggregation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.