Research map / Federated learning and privacy / Personalized and heterogeneous FL
Accelerating Heterogeneous Federated Learning with Closed-form Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fed3R, a federated closed-form ridge regression classifier on frozen pre-trained features that is invariant to client partition and sampling order, plus Fed3R-RF (random features) and Fed3R+FT (classifier initialization for fine-tuning).
Problem
Federated learning under strong statistical heterogeneity in cross-device settings suffers from client drift and classifier bias in the final layer, slowing convergence and hurting accuracy.
Approach
Federated version of recursive ridge regression: clients send exact local statistics (A_k = Z_k^T Z_k, b_k = Z_k^T Y_k) once, the server sums them and solves a closed-form one-vs-rest RR classifier, which equals the centralized solution regardless of data split; the RR solution is also used to initialize a softmax classifier for subsequent fine-tuning.
Key results
- Fed3R-RF with 10k random features reaches 56.6% on Landmarks and 47.6% on iNaturalist, matching the centralized RR solutions, while Fed3R (linear) reaches 49.6% and 45.1%.
- Fed3R-RF (D=10k) reaches 40% accuracy on Landmarks after 27.3 rounds versus 528.7 for FedAvg (about 19x speedup), with up to two orders of magnitude less communication and computation.
- Fed3R+FT feat variant reaches 67.4% on Landmarks (Scaffold init-based FT) and Fed3R+FT feat reaches 51.5% on iNaturalist with FedAvgM, outperforming baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 10k instead of 5k random features (ablation) | Fed3R-RF · Landmarks | accuracy: 53.9 -> 56.6 | Helped |
| Use 10k instead of 5k random features (ablation) | Fed3R-RF · iNaturalist | accuracy: 46.8 -> 47.6 | Helped |
| Replace the FedNCM classifier with Fed3R-RF 10k (ablation) | closed-form classifier · Landmarks | accuracy: 36.2 -> 56.6 | Helped |
| Replace the FedNCM classifier with Fed3R-RF 10k (ablation) | closed-form classifier · iNaturalist | accuracy: 32.2 -> 47.6 | Helped |
| Initialize the classifier with Fed3R (ablation) | FedAvg fine-tuning with classifier-only updates · iNaturalist | accuracy: 36.7 -> 42.0 | Helped |
| Initialize the classifier with Fed3R (ablation) | FedAvg fine-tuning of the whole model · iNaturalist | accuracy: 39.5 -> 49.0 | Helped |
| Initialize the classifier with Fed3R (ablation) | FedAvgM fine-tuning with classifier-only updates · iNaturalist | accuracy: 37.6 -> 43.5 | Helped |
| Initialize the classifier with Fed3R (ablation) | FedAvgM fine-tuning of the whole model · iNaturalist | accuracy: 39.3 -> 49.8 | Helped |
| Fine-tune only the feature extractor and keep the Fed3R classifier fixed instead of fine-tuning the whole model (ablation) | FedAvg fine-tuning initialized with Fed3R · iNaturalist | accuracy: 49.0 -> 50.8 | Helped |
| Fine-tune only the feature extractor and keep the Fed3R classifier fixed instead of fine-tuning the whole model (ablation) | FedAvgM fine-tuning initialized with Fed3R · iNaturalist | accuracy: 49.8 -> 51.5 | Helped |
| Fine-tune only the feature extractor and keep the Fed3R classifier fixed instead of fine-tuning the whole model (ablation) | FedAvg fine-tuning initialized with Fed3R · Landmarks | accuracy: 64.1 -> 59.6 | Hurt |
| Fine-tune only the feature extractor and keep the Fed3R classifier fixed instead of fine-tuning the whole model (ablation) | Scaffold fine-tuning initialized with Fed3R · Landmarks | accuracy: 67.4 -> 63.4 | Hurt |
Surprising findings
- Fixing the Fed3R classifier while fine-tuning only the feature extractor (Fed3R+FTfeat) gives better feature quality than fine-tuning the classifier too, on iNaturalist.
- On iNaturalist, the closed-form RR classifier at convergence beats the softmax classifier obtained after several fine-tuning strategies.
Limitations
- Fed3R (linear) can underperform gradient-based baselines on Landmarks when the pre-trained feature extractor is not adapted to the target task.
- Fed3R-RF has low computation but high communication cost due to the D^2 statistics, and Scaffold fails to converge on iNaturalist.
- Relies on a frozen pre-trained feature extractor; privacy relies on secure aggregation that is not implemented in experiments.
Experimental setup
Datasets: Landmark-Users-160K iNaturalist-Users-120K CIFAR-100 ImageNet-1k
Benchmarks: Landmark-Users-160K iNaturalist-Users-120K CIFAR-100
Metrics: accuracy rounds to target accuracy communication cost computation cost per client (GFLOPs)
Baselines: FedAvg FedAvgM Scaffold FedAvg-LP Scaffold-LP FedNCM FedDyn Mime
Builds on: FedAvg FedAvgM Scaffold FedNCM
Closest papers by meaning
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.89)
- FedUV: Uniformity and Variance for Heterogeneous Federated Learning (CVPR 2024 · similarity 0.88)
- SparsyFed: Sparse Adaptive Federated Learning (ICLR 2025 · similarity 0.88)
- FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices Using a Computing Power-Aware Scheduler (ICLR 2024 · similarity 0.87)
- FedEL: Federated Elastic Learning for Heterogeneous Devices (NeurIPS 2025 · similarity 0.87)
- Federated Learning with Sample-level Client Drift Mitigation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.