Research map / Federated learning and privacy / Personalized and heterogeneous FL

FLHetBench: Benchmarking Device and State Heterogeneity in Federated Learning

CVPR 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FLHetBench, a benchmark with Dirichlet-process-based device/state sampling and Monte Carlo heterogeneity metrics, and proposes BiasPrompt+ for federated learning under device and state heterogeneity.

Problem

Federated learning methods are rarely evaluated under realistic device capacity and client state availability heterogeneity, and existing statistical divergence metrics do not reflect FL performance under these constraints.

Approach

Dirichlet-process samplers (DPGMM for continuous device speeds, DPCSM for discrete client states) generate realistic device/state databases with controllable heterogeneity, paired with Monte Carlo participation metrics that correlate with FL performance; BiasPrompt+ combines gradient-surgery staleness-aware aggregation with ViT prompt and bias fast weights.

Key results

• DevMC-R reaches Pearson r=0.810 with FedAVG test accuracy on COVID-FL, versus near-zero or weak correlations for STD, KS, and JS.
• Under severe interplay heterogeneity (InterMC-R=0.56) on OpenImage, BiasPrompt+ achieves 35.75% test accuracy versus 20.78% for the best baseline (FetchSGD).
• InterMC-R and InterMC-T show strong correlations (r>0.84) with FL performance on COVID-FL and OpenImage.

Measured effects reported in the paper

Surprising findings

• Common straggler-mimicking strategies produce flat or decreasing training time as the straggler ratio increases, contrary to real-world expectations.
• Statistical divergence metrics (STD, KS, JS) correlate poorly with FL performance under device heterogeneity.

Limitations

• BiasPrompt+ still shows performance drops under highly heterogeneous conditions.
• Evaluation limited to ViT-B image classification on COVID-FL and OpenImage.

Experimental setup

Datasets: COVID-FL OpenImage ImageNet-1k FLASH state database (136k states) FL-Device (about 10,000 mobile devices)

Benchmarks: COVID-FL OpenImage

Metrics: test accuracy wall-clock training time DevMC-R DevMC-T StatMC-R StatMC-T InterMC-R InterMC-T Pearson correlation

Baselines: FedAVG FedProx FedKD FedDyn FetchSGD FedBuff

Builds on: FedAVG FLASH FedScale VPT prompt tuning

Closest papers by meaning

• FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.90)
• Tackling Intertwined Data and Device Heterogeneities in Federated Learning with Unlimited Staleness (AAAI 2025 · similarity 0.89)
• Federated Learning with Sample-level Client Drift Mitigation (AAAI 2025 · similarity 0.89)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity (CVPR 2024 · similarity 0.89)
• Benchmarking Algorithms for Federated Domain Generalization (ICLR 2024 · similarity 0.89)
• FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models (NeurIPS 2024 · similarity 0.88)