Research map / Label-efficient and robust learning / Test-time adaptation
Addressing Label Shift in Distributed Learning via Entropy Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VRLS, an entropy-regularized MLE density-ratio estimator for label shift, and embeds it in an importance-weighted ERM (IW-ERM) framework for multi-node distributed learning with intra- and inter-node label shifts.
Problem
Estimating test-to-train label density ratios under intra-node and inter-node label shift in distributed learning, where data must stay on each node and predictors are miscalibrated.
Approach
Adds a Shannon-entropy (negative softmax entropy) regularizer to the cross-entropy training of the predictor so its probabilities are better calibrated, then estimates label ratios by MLLS-style maximum likelihood on test data; these local ratios are aggregated across nodes and used as importance weights in IW-ERM.
Key results
- On Fashion MNIST with 5 nodes, Our IW-ERM reaches 0.7520 average accuracy vs. 0.5774 for the best baseline (SCAFFOLD) and 0.8273 for the true-ratio upper bound.
- On CIFAR-10 with 5 nodes, Our IW-ERM gets 0.5640 average accuracy vs. 0.4515 for FedAvg and 0.5790 for the upper bound.
- VRLS reports consistently lower ratio-estimation MSE than MLLS across Dirichlet alpha values and sample sizes on MNIST and CIFAR-10; the paper claims up to 20% gains in imbalanced settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use IW-ERM with VRLS instead of SCAFFOLD | distributed learning · Fashion MNIST, 5 nodes | average accuracy: 0.5774 ± 0.0036 -> 0.7520 ± 0.0209 | Helped |
| train the predictor using only 10% of the available training data (ablation) | Our IW-ERM on Fashion MNIST · Fashion MNIST, 5 nodes | average accuracy: 0.7520 ± 0.0209 -> 0.7376 ± 0.0099 | Hurt |
| use IW-ERM with VRLS instead of FedAvg | distributed learning · CIFAR-10, 5 nodes | average accuracy: 0.4515 ± 0.0148 -> 0.5640 ± 0.0241 | Helped |
| use IW-ERM with VRLS instead of FedAvg | distributed learning · CIFAR-10, 100 nodes | average accuracy: 0.3915 -> 0.5354 | Helped |
| use IW-ERM with VRLS instead of FedAvg | distributed learning · CIFAR-10, 200 nodes | average accuracy: 0.5942 -> 0.6216 | Helped |
Surprising findings
- FedBN achieves only 0.1537 and 0.1753 average accuracy with 100 and 200 nodes on CIFAR-10, far below its 5-node result.
Limitations
- Relies on a strict class-conditional assumption p(x|y) being identical across nodes and test sets.
- Relaxed label shift experiments are only preliminary and not validated on real-world datasets.
- Time complexity of IW-ERM is not reduced and is flagged as future work.
Experimental setup
Datasets: MNIST Fashion MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10 Fashion MNIST
Metrics: mean squared error of ratio estimates average node accuracy
Baselines: MLLS_L2 MLLS_EM FedAvg FedBN FedProx SCAFFOLD
Builds on: MLLS BBSE BCT IW-ERM FedAvg
Closest papers by meaning
- Regularized Learning for Domain Adaptation under Label Shifts (ICLR 2019 · similarity 0.86)
- Label Shift Meets Online Learning: Ensuring Consistent Adaptation with Universal Dynamic Regret (CVPR 2025 · similarity 0.86)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.86)
- Heterogeneous Label Shift: Theory and Algorithm (ICML 2025 · similarity 0.85)
- Label Noise: Ignorance Is Bliss (NeurIPS 2024 · similarity 0.84)
- Multi-Label Test-Time Adaptation with Bound Entropy Minimization (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.