Research map / Federated learning and privacy / Personalized and heterogeneous FL
Benchmarking Algorithms for Federated Domain Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Federated Domain Generalization benchmark with a new Heterogeneous Partitioning method that controls domain-based client heterogeneity, and evaluates 14 DG, FL, and federated DG methods on 7 datasets with 100 clients.
Problem
Existing evaluations of federated domain generalization are limited in client scale, dataset diversity, and do not control train-test and domain-based client heterogeneity; existing partitioning methods (shards, Dirichlet, semantic) fail to satisfy needed constraints.
Approach
A partition method (Heterogeneous Partitioning) that splits any multi-domain dataset across any number of clients, interpolating between homogeneous (lambda=1) and complete domain heterogeneity (lambda=0) while guaranteeing a true partition, plus a benchmark that applies centralized DG, FL, and federated DG methods under this partitioning.
Key results
- On PACS with C=100 clients, FedAvg-ERM reaches 0.95 at lambda=0 while most federated DG methods (FedADG 0.94, FedDG 0.94) do not clearly beat it, and FedSR/FedGMA drop to 0.54-0.72.
- On IWildCam, FedAvg-ERM accuracy/F1 falls from 0.34 centralized to 0.20 at C=100 with lambda=0, and most FL-DG methods underperform it.
- On CelebA, FedAvg-ERM drops from 0.77 (centralized) to 0.45 at lambda=0 with C=100, while only FedDG-like and AFL/Scaffold variants partly recover (AFL 0.83, Scaffold 0.80).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedAvg-ERM instead of the evaluated alternatives on general DG datasets | Federated DG methods · PACS and other general DG datasets | accuracy | Mixed |
| Move from homogeneous clients (λ=1) to heterogeneous clients (λ=0.1) | FedAvg-ERM · PACS | accuracy: 0.95±0.02 -> 0.96±0.02 | Helped |
| Move from homogeneous clients (λ=1) to heterogeneous clients (λ=0.1) | FedAvg-ERM · CelebA | accuracy: 0.61±0.02 -> 0.52±0.04 | Hurt |
| Use federated training rather than centralized training | Centralized DG methods · Image datasets | accuracy | Hurt |
| Use FL methods designed to tackle client heterogeneity | Federated DG methods · IWildCam and CivilComments | convergence | Helped |
| Use FedDG instead of FedAvg-ERM | Federated DG methods · CelebA | accuracy: 0.61±0.02 -> 0.67±0.00 | Helped |
| Use FedDG instead of ERM | Federated methods · PACS, λ=0 | accuracy: 0.95±0.02 -> 0.94±0.02 | Hurt |
| Increase the number of clients from C=1 to C=200 | Federated DG methods · PACS | accuracy: 90% to 10% | Hurt |
| Increase the number of clients | FedADG and FedSR · PACS | accuracy | Hurt |
| Increase communication rounds | Federated DG methods · PACS | accuracy | Mixed |
| Use a relatively small communication budget with early stopping | Federated DG methods · PACS | accuracy: 10 | Helped |
| Decrease λ, increasing client domain heterogeneity | Federated DG methods · IWildCam and Py150 | accuracy | Hurt |
Surprising findings
- FedAvg with ERM is a strong baseline that most federated DG methods fail to beat across datasets.
- Number of communication rounds does not monotonically affect DG performance; on PACS optimal performance occurs at about 10 rounds.
Limitations
- Federated DG methods still underperform FedAvg-ERM on most datasets at large client counts and high heterogeneity.
- Hyperparameter search is limited (8 grid-search runs per method) and some methods, such as Modified AFL, are modified from original designs.
- Some FL-DG methods like FedDG share amplitude information, raising privacy concerns.
Experimental setup
Datasets: FEMNIST PACS IWildCam CelebA Camelyon17 CivilComments Py150
Benchmarks: PACS IWildCam CelebA Camelyon17 CivilComments Py150 FEMNIST
Metrics: test accuracy macro F1
Baselines: FedAvg-ERM FedAdam IRM Fish Mixup MMD DeepCoral GroupDRO FedProx Scaffold AFL FedDG
Builds on: FedAvg DomainBed WILDS AFL GroupDRO
Closest papers by meaning
- Federated Domain Generalization with Data-free On-server Matching Gradient (ICLR 2025 · similarity 0.92)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.90)
- TTA-FedDG: Leveraging Test-Time Adaptation to Address Federated Domain Generalization (AAAI 2025 · similarity 0.90)
- FLHetBench: Benchmarking Device and State Heterogeneity in Federated Learning (CVPR 2024 · similarity 0.89)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.88)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.