Research map / Federated learning and privacy / Personalized and heterogeneous FL
Exploring Vacant Classes in Label-Skewed Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedVLS, a label-skewed federated learning method combining knowledge distillation restricted to vacant classes with a logit suppression penalty on non-label logits, outperforming prior SOTA on MNIST, CIFAR10/100, TinyImageNet and AG_news.
Problem
Label skews in federated learning cause local models to overfit, degrading minority-class accuracy and especially forgetting classes absent from a client's data (vacant classes), which prior class-balanced methods overlook.
Approach
Apply knowledge distillation from the global model only over vacant-class logits during local training to retain vacant-class knowledge, and add a logit suppression loss that penalizes non-label logits weighted by class prevalence to reduce minority-to-majority misclassification; combined with logit calibration (FedLC).
Key results
- On CIFAR10 with Dirichlet beta=0.05, FedVLS reaches 75.71% test accuracy, +12.81% over FedAvg (62.90%) and +18.40% over FedLC (57.31%).
- Quantity-based skew CIFAR10 (s=2): FedVLS 68.03% vs FedLMD 68.52% (best) and FedNTD 67.35%; FedVLS best on CIFAR100 (64.95%) and TinyImageNet (43.97%).
- Ablation on CIFAR10 (beta=0.05): logit suppression alone gives 70.25%, vacant-class distillation alone 71.53%, both 75.71%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add vacant-class distillation and logit suppression to the FedLC objective (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 57.31 -> 75.71 (+18.40) | Helped |
| Add logit suppression (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 57.31 -> 70.25 (+12.94) | Helped |
| Add vacant-class distillation (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 57.31 -> 71.53 (+14.22) | Helped |
| Add vacant-class distillation to logit suppression (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 70.25 -> 75.71 | Helped |
| Add logit suppression to vacant-class distillation (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 71.53 -> 75.71 | Helped |
| Change λ from 0.1 to 0.05 (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 75.71 -> 74.70 | Hurt |
| Change λ from 0.1 to 0.25 (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 75.71 -> 75.47 | Hurt |
| Change λ from 0.1 to 0.5 (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 75.71 -> 75.29 | Hurt |
| Change λ from 0.1 to 1 (ablation) | FedVLS · CIFAR10, Dirichlet β=0.05 | accuracy: 75.71 -> 74.98 | Hurt |
| Add FedVLS to FedLC | FedLC · TinyImageNet, Dirichlet β=0.05 | accuracy: 35.56 -> 45.23 (+9.67) | Helped |
| Add FedVLS to FedEXP | FedEXP · CIFAR10, Dirichlet β=0.05 | accuracy: 63.22 -> 75.80 (+12.58) | Helped |
| Add FedVLS to FedSAM | FedSAM · TinyImageNet, Dirichlet β=0.05 | accuracy: 41.14 -> 48.12 (+6.98) | Helped |
Surprising findings
- FedLC drops below FedAvg on CIFAR10 at beta=0.05 (57.31% vs 62.90%), attributed to many vacant classes per client.
- Vacant-class distillation outperforms distilling over all classes (FedKD, FedEKD) and not-true-class distillation (FedNTD) on vacant-class accuracy.
Limitations
- No theoretical analysis of convergence, privacy, or fairness (deferred to future work).
- Gains on quantity-based skew are not uniformly best (FedLMD slightly higher on CIFAR10 s=2).
- Majority-class accuracy can slightly decrease as minority/vacant classes improve.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 TinyImageNet AG_news Digits (MNIST, USPS, SVHN, SYN)
Benchmarks: MNIST CIFAR10 CIFAR100 TinyImageNet AG_news
Metrics: test accuracy (max global model accuracy, mean ± std over 3 class-wise accuracy drift diversity
Baselines: FedAvg FedProx MOON FedEXP FedLC FedRS FedSAM FedNTD FedMR FedLMD FedConcat FedGF
Builds on: FedAvg FedLC FedNTD FedLMD
Closest papers by meaning
- Exploiting Label Skews in Federated Learning with Model Concatenation (AAAI 2024 · similarity 0.91)
- (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning (NeurIPS 2024 · similarity 0.88)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.88)
- Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators (AAAI 2024 · similarity 0.87)
- Recurrent Early Exits for Federated Learning with Heterogeneous Clients (ICML 2024 · similarity 0.87)
- Federated Learning with Extremely Noisy Clients via Negative Distillation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.