Research map / Federated learning and privacy / Communication-efficient FL
Communication-Efficient Federated Group Distributionally Robust Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes communication-efficient federated group DRO algorithms (FGDRO-CVaR, FGDRO-KL, FGDRO-KL-Adam) that use compositional reformulations with moving-average estimators and local updates, achieving lower communication and sample complexity than prior federated GDRO methods.
Problem
Federated group distributionally robust optimization (FGDRO) under heterogeneous client data volumes and distributions suffers from high communication and sample complexity, since the weight vector over clients is only updated at the server.
Approach
Reformulate FGDRO with CVaR constraint as an unconstrained non-smooth two-level compositional problem with a scalar threshold s updated locally, and FGDRO-KL as a smooth three-level compositional problem; use moving-average estimators for the loss and global statistics so that local updates can use small batches, and extend to Adam-type local updates (FGDRO-KL-Adam).
Key results
- On Pile, FGDRO-KL-Adam reaches worst log-PPL 3.608 vs. 8.085 for FedAvg and 7.242 for FedAdam.
- On CivilComments, FGDRO-KL achieves the best average accuracy of 0.7734 and worst accuracy 0.6921 (FedAvg worst 0.6415).
- Theoretical communication complexity O(1/eps^4) for FGDRO-CVaR (sample O(1/eps^8)) and O(1/eps^3) for FGDRO-KL (sample O(1/eps^4)), versus O(1/eps^12) for DRFA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FGDRO-CVaR instead of FedAvg | GPT-2 · Pile test | worst log-perplexity: 8.085 -> 8.145 | Hurt |
| Use FGDRO-KL-Adam instead of FedAdam | GPT-2 · Pile test | worst log-perplexity: 7.242 -> 3.608 | Helped |
| Use FGDRO-KL instead of FedAdam | DistilBERT base-uncased · CivilComments test | worst group accuracy: 0.6567 -> 0.6921 | Helped |
| Use FGDRO-KL instead of FedProx | DistilBERT base-uncased · CivilComments test | average accuracy: 0.7692 -> 0.7734 | Helped |
| Use FGDRO methods instead of FedAdam | DenseNet-121 · Camelyon17 test | accuracy: 0.9493 -> 0.8667; 0.9493 -> 0.9243; 0.9493 -> 0.9399 | Hurt |
| Use FGDRO-KL instead of FedAvg | ResNet50 · iWildCam2020 test | Macro F1: 0.4964 -> 0.5201 | Helped |
| Use FGDRO-KL-Adam instead of FedProx | ResNet50 · PovertyMap test | worst Pearson: 0.7305 -> 0.7827 | Helped |
| Use FGDRO-KL-Adam instead of FedAdam | ResNet50 · PovertyMap test | average Pearson: 0.8273 -> 0.8225 | Hurt |
| Replace KL local SGD-type updates with Adam-type local updates | FGDRO-KL on Camelyon17 test · Camelyon17 test | accuracy: 0.9243 -> 0.9399 | Helped |
| Replace KL local SGD-type updates with Adam-type local updates | FGDRO-KL on iWildCam2020 test · iWildCam2020 test | Macro F1: 0.5201 -> 0.4489 | Hurt |
| Vary the communication interval (ablation) | FGDRO-CVaR and FGDRO-KL-Adam on Camelyon17 · Camelyon17 | performance | No clear effect |
| Use local Adam updates for ERM (LocalAdam) rather than FedAdam's global-only adaptive updates (ablation) | ERM on the experiments shown in Figure · Figure results | performance | Helped |
Surprising findings
- FGDRO-CVaR underperforms on average log-perplexity on Pile relative to FedAvg, showing a worst-case vs average trade-off.
- FedAdam achieves the best worst-case Pearson-free ordering in some cases; FedAdam leads on Camelyon17 accuracy (0.9493), above FGDRO variants.
Limitations
- No provable adaptive (Adam-type) variant for FGDRO-CVaR because subgradient moving averages for non-smooth objectives are not provably accurate.
- FGDRO-KL-Adam is not uniformly best; FedAdam outperforms it on Camelyon17 accuracy and FGDRO-CVaR underperforms on some metrics.
Experimental setup
Datasets: Pile CivilComments Camelyon17 iWildCam2020 PovertyMap CIFAR-10
Benchmarks: Pile CivilComments Camelyon17 iWildCam2020 PovertyMap
Metrics: worst log-perplexity average log-perplexity worst-group accuracy average accuracy accuracy macro F1 worst Pearson average Pearson
Baselines: FedAvg SCAFFOLD FedProx FedAdam DRFA DR-DSGD
Builds on: DRFA DR-DSGD FedAdam STORM
Closest papers by meaning
- Group Distributionally Robust Optimization with Flexible Sample Queries (NeurIPS 2026 · similarity 0.87)
- FedRTS: Federated Robust Pruning via Combinatorial Thompson Sampling (NeurIPS 2025 · similarity 0.86)
- Communication-Efficient Federated Learning with Accelerated Client Gradient (CVPR 2024 · similarity 0.85)
- Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization (ICLR 2025 · similarity 0.85)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.85)
- Byzantine-Robust Federated Learning: Impact of Client Subsampling and Local Updates (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.