Research map / Federated learning and privacy / Personalized and heterogeneous FL
Little Is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedCT, a federated co-training method where clients share only hard labels on a public unlabeled dataset and use majority-vote consensus as pseudo-labels, improving privacy over FedAvg and distributed distillation while enabling non-gradient models.
Problem
Federated learning and federated semi-supervised learning leak private information through shared parameters or soft labels, and restrict local models to gradient-trainable ones; the goal is to share less information while keeping model quality.
Approach
Clients iteratively share hard (one-hot class) predictions on a public unlabeled dataset; the server forms a majority-vote consensus that serves as pseudo-labels for local co-training. The approach also gives a convergence proof, a sensitivity bound for on-average-replace-one-stable learners, and DP via the XOR mechanism.
Key results
- On iid data with 5 clients, FedCT reaches vulnerability ~0.51-0.53 (near random guessing) versus ~0.71-0.76 for FedAvg, with comparable accuracy (e.g., FashionMNIST 0.84 vs 0.83).
- Communication on FashionMNIST is reduced by a factor of about 214 (≈12.2 KB vs ≈2.6 MB per round) relative to FedAvg.
- Fine-tuning GPT-2 on IMDB with 10 clients, FedCT reaches 0.73 test accuracy versus 0.59 for FedAvg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| share hard labels with consensus pseudo-labeling instead of model parameters | Federated training · Pneumonia | VUL: 0.76 -> 0.51 | Helped |
| share hard labels with consensus pseudo-labeling instead of soft labels | Federated training · Pneumonia | VUL: 0.63 -> 0.51 | Helped |
| share hard labels with consensus pseudo-labeling instead of model parameters | Federated training · FashionMNIST | ACC: 0.83\pm 0.024 -> 0.84\pm 0.004 | Helped |
| share hard labels with consensus pseudo-labeling instead of model parameters | Federated training · MRI | ACC: 0.66\pm 0.015 -> 0.64\pm 0.004 | Hurt |
| apply the XOR-mechanism for ε=0.1 differential privacy | FedCT · FashionMNIST | ACC: 0.84\pm 0.004 -> 0.80 \pm 0.001 | Hurt |
| use qualified majority instead of majority vote (ablation) | FedCT · Pathological non-iid α1=α2=0.01 | ACC: 0.6252 -> 0.7358 | Helped |
| use qualified majority instead of majority vote (ablation) | FedCT · Non-iid α1=100, α2=2 | ACC: 0.7975 -> 0.7968 | No clear effect |
| use non-iid CIFAR10 instead of iid CIFAR10 for public unlabeled data (ablation) | FedCT on CIFAR10 · CIFAR10 | ACC: 0.77 -> 0.74 | Hurt |
| use more dissimilar public unlabeled data (ablation) | FedCT on CIFAR10 · CIFAR10 | ACC: 0.77 -> 0.65 | Hurt |
| use FedCT instead of FedAvg for GPT2 fine-tuning | GPT2 federated fine-tuning · IMDB | ACC: 0.59 -> 0.73 | Helped |
| use FedCT instead of FedAvg for GPT2 fine-tuning | GPT2 federated fine-tuning · Twitter sentiment | ACC: 0.61 -> 0.65 | Helped |
| use a heterogeneous ensemble of model types instead of XGBoost on every client (ablation) | FedCT on BreastCancer · BreastCancer | ACC: 0.94 -> 0.95 | Helped |
Surprising findings
- On pathological non-iid data (alpha=0.01), plain majority-vote FedCT (0.625) underperforms FedAvg (0.735), while a qualified majority (0.9 quorum) recovers 0.736.
- Using only hard labels for the consensus gives a sensitivity ~3000 on FashionMNIST, yet DP-FedCT at epsilon=0.1 still achieves about 0.80 accuracy.
Limitations
- null
Experimental setup
Datasets: CIFAR10 FashionMNIST SVHN MRI brain tumor Pneumonia chest X-ray WineQuality Breastcancer AdultsIncome Mushroom Covertype IMDB Twitter sentiment
Benchmarks: CIFAR10 FashionMNIST SVHN MRI Pneumonia WineQuality BreastCancer AdultsIncome Mushroom Covertype IMDB Twitter
Metrics: test accuracy membership inference vulnerability (ROC AUC)
Baselines: FedAvg DP-FedAvg DD (distributed distillation) PATE DP-PATE FedMD
Builds on: FedAvg distributed distillation PATE co-training XOR mechanism
Closest papers by meaning
- (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning (NeurIPS 2024 · similarity 0.91)
- Federated Learning via Input-Output Collaborative Distillation (AAAI 2024 · similarity 0.89)
- FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.89)
- Progressive Distribution Matching for Federated Semi-Supervised Learning (AAAI 2025 · similarity 0.89)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.89)
- A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.