Research map / Federated learning and privacy / Personalized and heterogeneous FL
Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedU2, a federated unsupervised learning framework combining a flexible uniform regularizer (unbalanced optimal transport to a spherical Gaussian) and an efficient unified aggregator (multi-objective ADMM-based aggregation) to tackle representation collapse and inconsistent client representations under non-IID data.
Problem
Federated unsupervised learning with non-IID client data suffers from representation collapse (which propagates between local and global models) and inconsistent representation spaces across local models.
Approach
A uniformity regularizer that uses unbalanced optimal transport to push local representations toward a spherical Gaussian distribution without sharing data (FUR), paired with a server aggregator that formulates aggregation as multi-objective optimization over client model deviation change rates, solved via dual ADMM (EUA), so that all clients improve consistently.
Key results
- On CIFAR10 cross-silo (alpha=0.1), FedU2-BYOL reaches 85.58% linear probing and 86.24% FT 10%, versus 83.53% for Orchestra and 83.29% for FedBYOL in LP.
- On CIFAR100 cross-device (alpha=0.1), FedU2-SimCLR reaches 41.56% linear probing, above L-DAWA-SimCLR (41.28%) and FedSimCLR (37.09%).
- FedU2-BYOL achieves the best KNN accuracy across alpha in {0.1, 0.5, 5} on CIFAR10 (81.39, 82.21, 84.79) and CIFAR100 (53.87, 54.07, 55.06) in cross-silo settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FUR (ablation) | FedU2-BYOL · CIFAR10 cross-silo, α = 0.1 | KNN accuracy (%): 80.93 -> 81.39 | Helped |
| add EUA (ablation) | FedU2-BYOL · CIFAR10 cross-silo, α = 0.1 | KNN accuracy (%): 81.04 -> 81.39 | Helped |
| add FUR (ablation) | FedU2-BYOL · CIFAR10 cross-silo, α = 0.5 | KNN accuracy (%): 82.14 -> 82.21 | Helped |
| add EUA (ablation) | FedU2-BYOL · CIFAR10 cross-silo, α = 0.5 | KNN accuracy (%): 82.18 -> 82.21 | Helped |
| add FUR (ablation) | FedU2-BYOL · CIFAR10 cross-silo, α = 5 | KNN accuracy (%): 84.01 -> 84.79 | Helped |
| add EUA (ablation) | FedU2-BYOL · CIFAR10 cross-silo, α = 5 | KNN accuracy (%): 83.45 -> 84.79 | Helped |
| add FUR (ablation) | FedU2-BYOL · CIFAR100 cross-silo, α = 0.1 | KNN accuracy (%): 53.20 -> 53.87 | Helped |
| add EUA (ablation) | FedU2-BYOL · CIFAR100 cross-silo, α = 0.1 | KNN accuracy (%): 53.45 -> 53.87 | Helped |
| add FUR (ablation) | FedU2-BYOL · CIFAR100 cross-silo, α = 0.5 | KNN accuracy (%): 53.72 -> 54.07 | Helped |
| add EUA (ablation) | FedU2-BYOL · CIFAR100 cross-silo, α = 0.5 | KNN accuracy (%): 53.86 -> 54.07 | Helped |
| add FUR (ablation) | FedU2-BYOL · CIFAR100 cross-silo, α = 5 | KNN accuracy (%): 54.61 -> 55.06 | Helped |
| add EUA (ablation) | FedU2-BYOL · CIFAR100 cross-silo, α = 5 | KNN accuracy (%): 54.34 -> 55.06 | Helped |
Surprising findings
- FedDecorr, which decorrelates representations to avoid collapse, performs worst in the cross-device CIFAR100 setting, suggesting batch-statistics-based decorrelation is unsuitable for FUSL with small local data.
- Fully uniform distribution matching breaks alignment and deteriorates clustering, so the fitted singular values of FedU2 differ from pure Gaussian samples.
Limitations
- Experiments are limited to two small image benchmarks (CIFAR10/CIFAR100) with ResNet18 backbones.
- The convergence analysis relies on four mild but strong assumptions (smoothness, strong convexity, bounded variance and gradients), and convexity does not hold for deep networks.
- Hyperparameters such as lambda_U and rho were tuned on CIFAR10 and fixed for other settings; sensitivity to client count and local epochs is deferred to an appendix.
Experimental setup
Datasets: CIFAR10 CIFAR100
Benchmarks: CIFAR10 CIFAR100
Metrics: KNN accuracy linear probing accuracy fine-tuning accuracy with 1% and 10% labels
Baselines: FedSimSiam FedSimCLR FedBYOL FedU FedEMA FedX Orchestra L-DAWA FedDecorr
Builds on: SimCLR BYOL SimSiam FedAvg FedDecorr FedEMA FedU
Closest papers by meaning
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.87)
- Federated Model Heterogeneous Matryoshka Representation Learning (NeurIPS 2024 · similarity 0.87)
- Federated Representation Learning in the Under-Parameterized Regime (ICML 2024 · similarity 0.87)
- Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors (ICML 2024 · similarity 0.87)
- FedUV: Uniformity and Variance for Heterogeneous Federated Learning (CVPR 2024 · similarity 0.87)
- Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.