Research map / Label-efficient and robust learning / Semi-supervised learning
Robust Training of Federated Models with Extremely Label Deficiency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Twin-sight, a twin-model federated semi-supervised learning framework that trains a supervised and an unsupervised (SimCLR-style) model per client, linked by a neighborhood-preserving constraint to avoid gradient conflict under label deficiency.
Problem
Federated semi-supervised learning under label deficiency, where a single model trained with mixed supervised and unsupervised objectives on labeled and unlabeled clients suffers gradient conflict and client drift.
Approach
Decouple labeled and unlabeled learning into two models (a supervised model with CE or confidence-thresholded pseudo-labels and an unsupervised contrastive model), and align them with a neighborhood-preserving loss on batch Gram matrices so they co-guide each other.
Key results
- On CIFAR-10 with gamma=0.1, K=10 and 4 labeled / 6 unlabeled clients, Twin-sight reaches 70.06% accuracy versus 63.66% for the best baseline (FedProx-Lower Bound), reaching the target in 115 rounds.
- On SVHN and FMNIST (same setting), Twin-sight achieves 62.94% and 79.95%, outperforming baselines such as RSCFed (54.50%, 76.58%) and FedAvg+Freematch (59.74%).
- In the partially labeled scenario (5% labels per client), Twin-sight reaches 78.89% on CIFAR-10 and 73.24% on SVHN, the best among the compared methods, though it does not win on FMNIST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Twin-sight twin-model framework instead of the strongest baseline | Federated semi-supervised learning · CIFAR-10, γ=0.1, 4 labeled/6 unlabeled clients | accuracy: 63.66 -> 70.06 | Helped |
| Use the Twin-sight twin-model framework instead of the strongest baseline | Federated semi-supervised learning · CIFAR-100, γ=0.1, 4 labeled/6 unlabeled clients | accuracy: 48.73 -> 49.98 | Helped |
| Use the Twin-sight twin-model framework instead of the strongest baseline | Federated semi-supervised learning · SVHN, γ=0.1, 4 labeled/6 unlabeled clients | accuracy: 59.74 -> 62.94 | Helped |
| Use the Twin-sight twin-model framework instead of the strongest baseline | Federated semi-supervised learning · FMNIST, γ=0.1, 4 labeled/6 unlabeled clients | accuracy: 76.58 -> 79.95 | Helped |
| Use Twin-sight instead of the strongest baseline | Federated learning with 5% labeled data per client · CIFAR-10 | accuracy: 75.47 -> 78.89 | Helped |
| Use Twin-sight instead of the strongest baseline | Federated learning with 5% labeled data per client · SVHN | accuracy: 69.08 -> 73.24 | Helped |
| Use Twin-sight instead of the strongest baseline | Federated learning with 5% labeled data per client · CIFAR-100 | accuracy: 44.16 -> 45.62 | Helped |
| Use Twin-sight instead of the strongest baseline | Federated learning with 5% labeled data per client · FMNIST | accuracy: 81.09 -> 80.11 | Hurt |
| Increase data heterogeneity from γ=0.1 to γ=0.05 (ablation) | Twin-sight · CIFAR-10 and SVHN | accuracy: 70.06 -> 43.82; 62.94 -> 43.22 | Hurt |
| Replace SimCLR with BYOL or SimSiam as the self-supervised method (ablation) | Twin-sight on CIFAR-10 · CIFAR-10, γ=0.1 | performance | Hurt |
| Reduce the client sampling rate from 50% to 40% (ablation) | Twin-sight · CIFAR-10, γ=0.1 | accuracy: 70.06 -> 69.15 | Hurt |
| Increase the client pool from K=50 to K=100 (ablation) | Twin-sight · CIFAR-10, γ=0.1, α=60% | accuracy: 62.2 -> 56.78 | Hurt |
Surprising findings
- FedIRM produced NaN loss under severely skewed label distributions, so its results are reported as missing.
- RSCFed did not outperform Twin-sight despite being an FSSL-specific method, and simple FedAvg+FixMatch/Freematch combinations were competitive in some settings.
Limitations
- Twin-model paradigm doubles the model per client, increasing memory and communication overhead.
- Performance on FMNIST in the partially labeled scenario does not beat the best baseline.
- Experiments use only small image datasets with ResNet-18 at 32x32 resolution.
Experimental setup
Datasets: CIFAR-10 SVHN Fashion-MNIST CIFAR-100
Benchmarks: CIFAR-10 SVHN FMNIST CIFAR-100
Metrics: accuracy communication rounds to reach target accuracy
Baselines: FedAvg-Lower Bound FedProx-Lower Bound FedAvg+FixMatch FedProx+FixMatch FedAvg+Freematch FedProx+Freematch Fed-Consist FedIRM RSCFed FedSem FedSiam FedMatch
Builds on: FedAvg FixMatch SimCLR FedIRM RSCFed
Closest papers by meaning
- (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning (NeurIPS 2024 · similarity 0.88)
- Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators (AAAI 2024 · similarity 0.86)
- Mind the Gap: Confidence Discrepancy Can Guide Federated Semi-Supervised Learning Across Pseudo-Mismatch (CVPR 2025 · similarity 0.86)
- SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning (AAAI 2025 · similarity 0.86)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.86)
- BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.