Research map / Federated learning and privacy / Personalized and heterogeneous FL
Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InCo Aggregation, which uses normalized internal cross-layer gradients (shallow plus deep layer gradients in the server model, with a closed-form convex projection) to extend model-homogeneous FL methods to heterogeneous-architecture FL.
Problem
Federated learning under system heterogeneity, where clients hold different model sizes; heterogeneous-model FL underperforms homogeneous FL because deep-layer representations diverge across clients.
Approach
Mix shallow-layer gradients into deep-layer gradients inside the server model (no extra communication), normalize them, and apply a closed-form projection enforcing non-negative alignment with the shallow gradient, so model-homogeneous methods (FedAvg, FedProx, FedNova, Scaffold, MOON) can be extended to heterogeneous models.
Key results
- InCoAvg improves HeteroAvg on CIFAR-10 (alpha=0.5) from 64.8 to 67.8 and on CINIC-10 (alpha=0.5) from 48.6 to 53.0 with stage splitting ResNets.
- InCo improves HeteroFL, InclusiveFL, FedRolex and ScaleFL by roughly 0.6 to 3.4 points across datasets without added communication overhead.
- InCo adds only about 0.4M FLOPs, computed on the server side.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply InCo Aggregation to model-homogeneous FL baselines | FedAvg, FedProx, FedNova, Scaffold, and MOON with heterogeneous models · Fashion-MNIST, SVHN, CIFAR-10, and CINIC-10; ResNet and ViT | test accuracy: ↑2.4; ↑2.4; ↑2.5; ↑2.1; ↑3.0; ↑4.0; ↑4.4; ↑1.0 | Helped |
| Apply InCo Aggregation to model-heterogeneous methods | HeteroFL, InclusiveFL, FedRolex, and ScaleFL · Fashion-MNIST, SVHN, and CIFAR-10; ResNet | test accuracy: ↑1.1; ↑0.7; ↑1.6; ↑1.3; ↑3.0; ↑2.8 | Helped |
| Remove gradient normalization (ablation) | InCoAvg · Heterogeneous FL | test accuracy | Hurt |
| Remove convex optimization (ablation) | InCoAvg · Heterogeneous FL | test accuracy | Hurt |
| Remove both gradient normalization and convex optimization (ablation) | InCoAvg · Heterogeneous FL | test accuracy | Hurt |
Surprising findings
- Shallow-layer gradients are more cross-client similar than deep-layer gradients, and even the lowest shallow-layer similarity exceeds the highest deep-layer similarity.
Limitations
- Smallest sub-models under InCo show lower CKA similarity to other clients, which the authors flag as unresolved.
- Convergence analysis relies on strong assumptions (bounded expected gradient norms and covariance bounds).
Experimental setup
Datasets: Fashion-MNIST SVHN CIFAR-10 CINIC-10
Benchmarks: Fashion-MNIST SVHN CIFAR-10 CINIC-10
Metrics: test accuracy communication overhead FLOPs CKA layer similarity
Baselines: HeteroFL InclusiveFL FedRolex ScaleFL AllSmall AllLarge HeteroAvg HeteroProx
Builds on: FedAvg FedProx FedNova Scaffold MOON
Closest papers by meaning
- A Swiss Army Knife for Heterogeneous Federated Learning: Flexible Coupling via Trace Norm (NeurIPS 2024 · similarity 0.89)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.88)
- Modeling Inter-Intra Heterogeneity for Graph Federated Learning (AAAI 2025 · similarity 0.88)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.87)
- An Aggregation-Free Federated Learning for Tackling Data Heterogeneity (CVPR 2024 · similarity 0.87)
- Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.