Research map / Federated learning and privacy / Personalized and heterogeneous FL
Subspace Constraint and Contribution Estimation for Heterogeneous Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedSCE, a heterogeneous federated learning method that constrains the update of a projection layer of local models to a dynamically updated low-rank subspace and weights auxiliary-model aggregation by feature- and parameter-space update distances.
Problem
Heterogeneous federated learning with auxiliary models suffers from local overfitting/catastrophic forgetting during local training and aggregation bias when combining auxiliary models across heterogeneous clients under data skew.
Approach
Restrict the update of specific (final) local model layers to a low-rank subspace refreshed via truncated SVD of recent update trajectories, combined with deep mutual learning between local and auxiliary models, and weight client auxiliary models by their feature- and parameter-space update distances instead of simple averaging.
Key results
- On Office10, FedSCE reaches 84.55% client-level... data-level average accuracy 76.34% vs 73.26% for the best baseline (about +3.08% D.Avg, +3.87% reported in abstract).
- On Digit5, FedSCE achieves 93.69% D.Avg, slightly above FedKD's 93.62%.
- Ablation on Office10: subspace constraint alone raises D.Avg from 73.26% to 73.90%, deep mutual learning to 75.31%, and contribution estimation to 76.34%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add local subspace learning to SOLO (ablation) | FedSCE local training · Office10 | C.Avg accuracy: 78.66 -> 79.93 | Helped |
| Add local subspace learning to SOLO (ablation) | FedSCE local training · Office10 | D.Avg accuracy: 73.26 -> 73.90 | Helped |
| Add local deep mutual learning (DML) to LSL (ablation) | FedSCE local training · Office10 | D.Avg accuracy: 73.90% to 75.31% | Helped |
| Add auxiliary model contribution estimation (CE) to LSL and DML (ablation) | FedSCE · Office10 | C.Avg and D.Avg accuracy: +1.02% and 1.03% | Helped |
| Vary projection-loss coefficient μ across 0.1, 1, 5, and 10 (ablation) | FedSCE · Office10 and CIFAR-10 | accuracy: Office10: 76.97, 76.81, 76.34, 76.96; Cifar10: 72.89, 72.67, 72.86, 72.80 | No clear effect |
| Increase local epochs from E=1 to E=10 and E=20 (ablation) | FedSCE · Office10 | D.Avg accuracy: 70.79 -> 77.20 -> 78.31 | Helped |
| Vary the number of constrained layers p (ablation) | FedSCE · Office10 | accuracy: 76.81 | Mixed |
| Vary subspace rank R (ablation) | FedSCE · Office10 | accuracy | No clear effect |
| Use FedSCE instead of the strongest baseline | Heterogeneous FL · Tiny-ImageNet, Pat | accuracy: 30.76 -> 30.49 | Hurt |
| Use FedSCE instead of the strongest baseline | Heterogeneous FL · Tiny-ImageNet, Dir0.1 | accuracy: 28.42 -> 28.38 | Hurt |
| Use FedSCE instead of the strongest baseline | Heterogeneous FL · Office10, D.Avg | accuracy: 73.26 -> 76.34 | Helped |
Surprising findings
- Simply adding the projection constraint without federated techniques already improves local-only accuracy (SOLO 73.26% to 73.90% D.Avg on Office10).
Limitations
- Subspace constraint applied only to final layers; hyperparameters (p, R, mu) require tuning and performance is non-monotonic in R.
- Baseline comparisons use a single shared auxiliary model (smallest architecture) and small-scale image benchmarks only.
Experimental setup
Datasets: Digit5 Office10 Cifar10 Cifar100 Tiny-Imagenet
Benchmarks: Digit5 Office10 Cifar10 Cifar100 Tiny-Imagenet
Metrics: D.Avg accuracy C.Avg accuracy
Baselines: SOLO FML FedKD FedDistill FedProto FedTGP
Builds on: FedAvg deep mutual learning subspace learning FML FedKD
Closest papers by meaning
- An Aggregation-Free Federated Learning for Tackling Data Heterogeneity (CVPR 2024 · similarity 0.88)
- Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.88)
- Federated Model Heterogeneous Matryoshka Representation Learning (NeurIPS 2024 · similarity 0.87)
- Adaptive Latent-Space Constraints in Personalized Federated Learning (NeurIPS 2025 · similarity 0.87)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.87)
- FedImpro: Measuring and Improving Client Update in Federated Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.