Research map / Continual learning / Online and replay-based
Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Federated Orthogonal Training (FOT), which extracts global principal activation subspaces per layer via privacy-preserving randomized SVD and projects aggregated federated updates orthogonal to old-task subspaces to mitigate global catastrophic forgetting in continual federated learning.
Problem
Continual federated learning suffers from global catastrophic forgetting: when a global model is trained on new tasks arriving at clients, its accuracy on old tasks drops, while clients cannot store old data and must keep privacy (secure aggregation compatible).
Approach
Clients train normally; the server projects aggregated updates onto the orthogonal complement of the global principal input subspace of previous tasks (FedProject). The subspace is extracted once per task via a GPSE round in which clients send random Gaussian sketches of projected layer inputs, summed with secure aggregation and decomposed by SVD, adapting GPM to the federated setting.
Key results
- FOT reaches 90.35% ACC and 1.75% FGT on IID PMNIST, versus 89.81% / 4.84% for the best baseline RGO+FL.
- On IID Split-CIFAR100, FOT achieves 71.90% ACC with 0.87% forgetting, versus 65.42% ACC for the best baseline ER+FL.
- On IID Split Mini-ImageNet, FOT achieves 69.07% ACC and 0.19% FGT, versus 59.26% ACC for GLFC; the abstract claims up to 15% accuracy gain and 27% lower forgetting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FOT instead of the strongest listed baseline | CFL · PMNIST IID | ACC (%) | Helped |
| Use FOT instead of the strongest listed baseline | CFL · CIFAR100 IID | ACC (%) | Helped |
| Use FOT instead of the strongest listed baseline | CFL · 5-Datasets IID | ACC (%) | Helped |
| Use FOT instead of the strongest listed baseline | CFL · Mini-Imagenet IID | ACC (%) | Helped |
| Use FOT instead of the strongest listed baseline | CFL · PMNIST non-IID | ACC (%) | Helped |
| Use FOT instead of the strongest listed baseline | CFL · CIFAR100 non-IID | ACC (%) | Helped |
| Use FOT instead of the strongest listed baseline | CFL · 5-Datasets non-IID | ACC (%) | Helped |
| Use FOT instead of the strongest listed baseline | CFL · Mini-Imagenet non-IID | ACC (%) | Helped |
| Apply Gaussian mechanism in the GPSE round (ablation) | FOT · Split Mini-Imagenet IID | ACC (%) | Hurt |
| Freeze the first layer after the first task (ablation) | FOT · Split Mini-Imagenet IID and non-IID | ACC (%) | No clear effect |
Surprising findings
- Federated ER improves over baselines by only 0.48% versus 2.71% in the centralized setting, suggesting old-task information is poorly preserved globally in FL.
- Applying GPM locally per client degrades the global model in non-IID settings (e.g., PMNIST non-IID ACC 72.17% vs FedAvg 80.06%).
Limitations
- Assumes known, well-separated task boundaries in the task-incremental setting.
- Applying a Gaussian mechanism for formal DP reduces performance (e.g., Mini-ImageNet IID ACC drops to 66.77%).
Experimental setup
Datasets: Permuted MNIST MNIST Split-CIFAR100 Split Mini-ImageNet 5-Datasets (CIFAR-10, MNIST, SVHN, notMNIST, FashionMNIST)
Benchmarks: Permuted MNIST Split-CIFAR100 5-Datasets Split Mini-ImageNet
Metrics: ACC FGT (forgetting)
Baselines: FedAvg (FL) EWC+FL ER+FL RGO+FL GPM+FL GLFC FedCIL
Builds on: GPM FedAvg EWC ER RGO
Closest papers by meaning
- Accurate Forgetting for Heterogeneous Federated Continual Learning (ICLR 2024 · similarity 0.91)
- Federated Continual Learning Goes Online: Uncertainty-Aware Memory Management for Vision Tasks and Beyond (ICLR 2025 · similarity 0.89)
- Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting (ICLR 2025 · similarity 0.89)
- FedSSI: Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence (ICML 2025 · similarity 0.88)
- FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning (CVPR 2024 · similarity 0.88)
- Handling Spatial-Temporal Data Heterogeneity for Federated Continual Learning via Tail Anchor (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.