Research map / Federated learning and privacy / Personalized and heterogeneous FL

A Mutual Information Perspective on Federated Contrastive Learning

ICLR 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Non-IID federated learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives a federated SimCLR objective from a multi-view mutual information view by adding a user-verification (client classification) loss to local SimCLR, plus label-dependent semi-supervised variants, and evaluates on CIFAR-10/100 and TinyImageNet under label skew, covariate shift, and joint shift.

Problem

Federated unsupervised and semi-supervised contrastive learning: the global multi-view mutual information does not decompose into client-wise losses, so standard FedAvg-compatible objectives are unclear, and different sources of non-i.i.d.-ness affect whether a global objective helps.

Approach

Decompose global multi-view MI into local conditional MI plus client-ID MI terms; bound them with InfoNCE and client classifiers (user-verification loss) so each client runs local SimCLR while training a shared client classifier, and extend to label-dependent contrastive losses plus label classifiers for semi-supervised learning.

Key results

• Federated SimCLR improves over Local SimCLR under label skew on CIFAR-10 (85.0% vs 79.4%) and CIFAR-100 (48.5% vs 42.2%) in the unsupervised setting.
• Local SimCLR outperforms federated SimCLR under covariate shift on CIFAR-10 (74.3% vs 73.8%) and CIFAR-100 (41.2% vs 39.5%), consistent with the theory that the global objective hurts when labels are independent of client ID.
• In the semi-supervised setting with 10% labels, Federated SimCLR reaches 78.0% vs 74.5% for Local SimCLR on CIFAR-10 label skew.

Measured effects reported in the paper

Surprising findings

• Adding the user-verification loss to SimSiam degraded unsupervised performance (e.g., 40.0% to 35.4% on CIFAR-10 label skew), and SimSiam + UV was at 0.5% on TinyImageNet unsupervised joint and covariate shift.
• Adding FeatARC alignment regularization did not improve accuracy and diverged at lambda=1.0 in the authors' cross-device setting.

Limitations

• The user-verification loss becomes problematic with many local updates since each client sees only one class, requiring frequent synchronization.
• Under covariate shift the global (UV) objective is detrimental, so the method must be chosen according to the type of non-i.i.d.-ness.
• Experiments are limited to image benchmarks with simulated partitions and a ResNet18 or CCT encoder.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 TinyImageNet

Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet

Metrics: linear probe test accuracy

Baselines: Local SimCLR Spectral CL SimSiam FeatARC SemiFed CBAFed FedCA FedMoCo FedEMA Hetero-SSFL

Builds on: SimCLR InfoNCE FedAvg spectral contrastive learning SimSiam

Closest papers by meaning

• Relaxed Contrastive Learning for Federated Learning (CVPR 2024 · similarity 0.91)
• FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data (ICML 2024 · similarity 0.87)
• (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning (NeurIPS 2024 · similarity 0.85)
• FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning (AAAI 2024 · similarity 0.85)
• Personalized Label Inference Attack in Federated Transfer Learning via Contrastive Meta Learning (AAAI 2025 · similarity 0.84)
• Federated Generalized Category Discovery (CVPR 2024 · similarity 0.84)