Research map / Federated learning and privacy / Communication-efficient FL
On Sampling Strategies for Spectral Model Sharding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes two optimal sampling strategies (Unbiased Horvitz-Thompson and Collective estimator) for spectral model sharding in heterogeneous federated learning, with learning-rate clipping, and shows gains over Top-n and PriSM on CIFAR-10/100, TinyImageNet and Shakespeare.
Problem
Heterogeneous federated clients with limited compute must train low-rank sub-models obtained by SVD sharding; existing randomized sampling (PriSM) is heuristic and its statistical properties are unclear, while deterministic top-n sampling (FedHM) is biased in the exploitative direction.
Approach
Derive inclusion probabilities for SVD terms in closed form by minimizing expected squared Frobenius approximation error under either an unbiasedness constraint (Horvitz-Thompson weights 1/pi_i) or a collective-averaging constraint across clients in a round; add learning-rate clipping for large auxiliary multipliers and sample with conditional Poisson sampling.
Key results
- On CIFAR-10 with ResNet at keep ratio 0.1, Collective reaches 82.59% vs 67.44% for Top-n and 71.15% for PriSM.
- On CIFAR-100 with CCT at keep ratio 0.1, Collective reaches 49.93% vs 47.34% for Top-n; Unbiased reaches 45.01%.
- Removing learning-rate clipping drops Unbiased accuracy on CIFAR-100 ResNet at r=0.1 from 35.12% to 30.14%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Collective sampling instead of Top-n | Spectral model sharding · CIFAR-10 ResNet, keep ratio 0.1 | accuracy: 67.44 ± 2.34 -> 82.59 ± 0.27 | Helped |
| Use Collective sampling instead of Top-n | Spectral model sharding · TinyImagenet CCT, keep ratio 0.1 | accuracy: 34.29 ± 0.12 -> 38.72 ± 0.86 | Helped |
| Use Collective sampling instead of Top-n | Spectral model sharding · CIFAR-100 ResNet, keep ratio 0.1 | accuracy: 21.13 ± 2.01 -> 37.83 ± 1.24 | Helped |
| Use Collective sampling instead of Top-n | Spectral model sharding · CIFAR-100 CCT, keep ratio 0.1 | accuracy: 47.34 ± 1.10 -> 49.93 ± 1.77 | Helped |
| Use Collective sampling instead of Top-n | Spectral model sharding · Shakespeare Transformer, keep ratio 0.1 | accuracy: 48.83 ± 0.45 -> 48.64 ± 0.50 | No clear effect |
| Add effective learning-rate clipping (ablation) | Unbiased spectral sharding · CIFAR-100 ResNet, keep ratio 0.1 | accuracy: 30.14 ± 1.58 -> 35.12 ± 1.51 | Helped |
| Add effective learning-rate clipping (ablation) | Unbiased spectral sharding · CIFAR-100 CCT, keep ratio 0.1 | accuracy: 32.46 ± 0.71 -> 45.01 ± 1.07 | Helped |
| Add effective learning-rate clipping (ablation) | Unbiased spectral sharding · CIFAR-100 ResNet, keep ratio 0.2 | accuracy: 43.11 ± 1.02 -> 46.34 ± 0.02 | Helped |
| Add effective learning-rate clipping (ablation) | Unbiased spectral sharding · CIFAR-100 CCT, keep ratio 0.2 | accuracy: 46.67 ± 0.85 -> 53.39 ± 0.81 | Helped |
| Scale sampled terms to preserve the full matrix Frobenius norm (ablation) | Top-n spectral sharding · CIFAR-10 ResNet, keep ratio 0.1 | accuracy: 67.44 ± 2.34 -> 63.01 ± 0.65 | Hurt |
| Replace Frobenius norm with Schatten norm (p=4) (ablation) | Unbiased spectral sharding · CIFAR-100 ResNet | accuracy: 35.12 ± 1.51 -> 34.54 ± 0.62 | No clear effect |
| Replace CPS sampling with Brewer or Minimum support while preserving marginal inclusion probabilities (ablation) | Unbiased and Collective sampling · CIFAR-100 ResNet and CCT, keep ratios 0.1 and 0.2 | accuracy | No clear effect |
Surprising findings
- Top-n sampling with Frobenius-norm scaling often hurts performance (e.g., CIFAR-10 ResNet r=0.1 drops from 67.44% to 63.01%).
- Unbiasedness applied to PriSM helps ResNet but is less helpful or harmful for attention-based architectures.
Limitations
- Gains over baselines are not significant on Shakespeare, where Top-n is competitive.
- Collective and Unbiased strategies have slower convergence than baselines due to more explorative sampling.
- Schatten-norm extension gave no significant improvement and numerical optimization was unstable for large p.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet Shakespeare
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet Shakespeare
Metrics: test accuracy average normalized marginal entropy
Baselines: Top-n (FedHM) PriSM PriSM + Scaled PriSM + Wallenius PriSM + Wallenius + ClipLR No sharding Top-n + Scaled
Builds on: FedHM PriSM FedAvg Eckart-Young-Mirsky theorem Conditional Poisson sampling
Closest papers by meaning
- Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning (NeurIPS 2024 · similarity 0.85)
- FedRTS: Federated Robust Pruning via Combinatorial Thompson Sampling (NeurIPS 2025 · similarity 0.84)
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.84)
- Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable (NeurIPS 2025 · similarity 0.84)
- FedUV: Uniformity and Variance for Heterogeneous Federated Learning (CVPR 2024 · similarity 0.84)
- Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysis (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.