Research map / Federated learning and privacy / Personalized and heterogeneous FL
Relaxed Contrastive Learning for Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedRCL, a federated learning method that applies a relaxed supervised contrastive loss (with a divergence penalty on overly similar same-class pairs) at all intermediate layers to reduce local gradient deviation and representation collapse.
Problem
Data heterogeneity and class imbalance in federated learning cause inconsistent local updates and slow convergence; naive supervised contrastive learning in FL causes representation collapse and limited transferability.
Approach
Derives a sample-wise deviation bound showing supervised contrastive learning mitigates local gradient deviation, then replaces SCL with a relaxed contrastive loss that penalizes excessively similar same-class pairs (above a threshold) and applies it across all intermediate layers, without needing global models or prototypes.
Key results
- On CIFAR-100 with alpha=0.05, FedRCL reaches 54.63% at 1000 rounds versus 39.86% for FedAvg (100 clients, 5% participation).
- On CIFAR-10 with alpha=0.05, FedRCL reaches 76.74% at 1000 rounds versus 63.42% for FedAvg.
- Integrating FedRCL with FedACG improves CIFAR-100 accuracy (alpha=0.3) from 62.51% to 66.38%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FedRCL instead of the strongest listed baseline, FedDecorr | federated ResNet-18 · CIFAR-10, α=0.05, 1000R | accuracy: 66.62 -> 76.74 | Helped |
| use FedRCL instead of the strongest listed baseline, FedMLB | federated ResNet-18 · CIFAR-100, α=0.05, 1000R | accuracy: 43.89 -> 54.63 | Helped |
| use FedRCL instead of the strongest listed baseline, FedNTD | federated ResNet-18 · Tiny-ImageNet, α=0.05, 1000R | accuracy: 28.96 -> 34.60 | Helped |
| use FedRCL instead of the strongest listed baseline, MOON | federated ResNet-18 · CIFAR-10, α=0.3, 1000R | accuracy: 86.30 -> 88.44 | Helped |
| use FedRCL instead of the strongest listed baseline, MOON | federated ResNet-18 · CIFAR-100, α=0.3, 1000R | accuracy: 58.37 -> 63.46 | Helped |
| use FedRCL instead of the strongest listed baseline, FedMLB | federated ResNet-18 · Tiny-ImageNet, α=0.3, 1000R | accuracy: 40.16 -> 44.95 | Helped |
| replace relaxed contrastive loss with supervised contrastive loss (ablation) | FedRCL · CIFAR-100, α=0.05, 1000R | accuracy: 54.63 -> 42.93 | Hurt |
| replace relaxed contrastive loss with supervised contrastive loss (ablation) | FedRCL · CIFAR-100, α=0.1, 1000R | accuracy: 58.23 -> 48.09 | Hurt |
| replace relaxed contrastive loss with supervised contrastive loss (ablation) | FedRCL · CIFAR-100, α=0.3, 1000R | accuracy: 63.46 -> 51.70 | Hurt |
| replace relaxed contrastive loss with self-supervised contrastive loss (ablation) | FedRCL · CIFAR-100, α=0.05, 1000R | accuracy: 54.63 -> 41.45 | Hurt |
| replace relaxed contrastive loss with self-supervised contrastive loss (ablation) | FedRCL · CIFAR-100, α=0.1, 1000R | accuracy: 58.23 -> 45.91 | Hurt |
| replace relaxed contrastive loss with self-supervised contrastive loss (ablation) | FedRCL · CIFAR-100, α=0.3, 1000R | accuracy: 63.46 -> 50.16 | Hurt |
Surprising findings
- Naive SCL integration into FedAvg is slower than baseline early in training (FedSCL 21.22% at 500 rounds vs 31.20% for FedAvg on CIFAR-100, alpha=0.05).
- FedDecorr, which mitigates dimensional collapse, shows degraded performance in some settings.
Limitations
- Gains are reported on small-scale image benchmarks with ResNet-18 only.
- The relaxed loss adds a threshold lambda and weight beta that require tuning.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: test accuracy at 500 and 1000 rounds effective rank variability collapse index within-class and between-class variance
Baselines: FedAvg FedAvg + FitNet FedProx MOON FedMLB FedLC FedNTD FedProc FedDecorr FedSCL FedCL FedAvgM
Builds on: FedAvg supervised contrastive learning FedLC MOON
Closest papers by meaning
- A Mutual Information Perspective on Federated Contrastive Learning (ICLR 2024 · similarity 0.91)
- FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning (AAAI 2024 · similarity 0.88)
- FedCS: Coreset Selection for Federated Learning (CVPR 2025 · similarity 0.87)
- Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation (NeurIPS 2025 · similarity 0.86)
- FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data (ICML 2024 · similarity 0.86)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.