Research map / Federated learning and privacy / Communication-efficient FL
Low Precision Local Training is Enough for Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes low-precision local training in federated learning (block floating point quantization of clients' training and communication) with high-precision moving-average aggregation on the server, showing 8-bit models match full-precision FL and reduce client overfitting.
Problem
Federated learning clients are resource-constrained and communicate over low-bandwidth networks, so local training is costly in computation and communication, and long local training causes client drift under heterogeneous data.
Approach
Quantize all local training computations (weights, activations, gradients, momentum) and client-to-server communication to low precision via block floating point, while the server performs full-precision averaging plus a moving average over the received low-precision models to recover high-precision accuracy.
Key results
- With FedAvg, 8-bit low-precision training with moving average matches or exceeds full precision across datasets and heterogeneity levels (e.g., CIFAR10 alpha=0.01: 53.3% at 8 bit vs 52.7% at 32 bit).
- Without moving average, 6-bit training drops sharply (e.g., CIFAR100 alpha=0.01: 3.16% vs 12.5% at 32 bit), showing the averaging's importance.
- Compared with HeteroFL and SplitMix at similar memory budgets, 5-8 bit FedAvg achieves higher accuracy (e.g., 53.3% at 8 bit vs 41.1% for HeteroFL full model) with lower training cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 8-bit local training with server moving average instead of 32-bit FedAvg | FedAvg · FashionMNIST, α = 0.01 | accuracy: 80.7 ± 0.6 -> 80.1 ± 0.7 | No clear effect |
| Use 8-bit local training with server moving average instead of 32-bit FedAvg | FedAvg · CIFAR10, α = 0.01 | accuracy: 52.7 ± 0.6 -> 53.3 ± 2.7 | No clear effect |
| Use 8-bit local training with server moving average instead of 32-bit FedAvg | FedAvg · CINIC10, α = 0.01 | accuracy: 41.0 ± 0.8 -> 43.3 ± 1.2 | Helped |
| Use 8-bit local training with server moving average instead of 32-bit FedAvg | FedAvg · CIFAR100, α = 0.01 | accuracy: 15.6 ± 0.0 -> 15.2 ± 0.2 | No clear effect |
| Lower local training precision from 8 bits to 6 bits (ablation) | FedAvg with server moving average · FashionMNIST, α = 0.01 | accuracy: 80.1 ± 0.7 -> 78.0 ± 1.4 | Hurt |
| Lower local training precision from 8 bits to 6 bits (ablation) | FedAvg with server moving average · CIFAR10, α = 0.01 | accuracy: 53.3 ± 2.7 -> 49.3 ± 0.9 | Hurt |
| Lower local training precision from 8 bits to 6 bits (ablation) | FedAvg with server moving average · CINIC10, α = 0.01 | accuracy: 43.3 ± 1.2 -> 39.2 ± 1.5 | Hurt |
| Lower local training precision from 8 bits to 6 bits (ablation) | FedAvg with server moving average · CIFAR100, α = 0.01 | accuracy: 15.2 ± 0.2 -> 12.4 ± 0.0 | Hurt |
| Add server moving average to 8-bit FedAvg (ablation) | FedAvg · FashionMNIST, α = 0.01 | accuracy: 72.2 ± 2.0 -> 80.1 ± 0.7 | Helped |
| Add server moving average to 8-bit FedAvg (ablation) | FedAvg · CIFAR10, α = 0.01 | accuracy: 25.6 ± 1.4 -> 53.3 ± 2.7 | Helped |
| Add server moving average to 8-bit FedAvg (ablation) | FedAvg · CINIC10, α = 0.01 | accuracy: 20.0 ± 1.2 -> 43.3 ± 1.2 | Helped |
| Add server moving average to 8-bit FedAvg (ablation) | FedAvg · CIFAR100, α = 0.01 | accuracy: 7.32 ± 0.4 -> 15.2 ± 0.2 | Helped |
Surprising findings
- Low-precision local training relieves client overfitting and reduces client drift, lowering test loss on the server.
- Quantized 8-bit models with moving-average aggregation can outperform full-precision FedAvg in several settings.
Limitations
- Real acceleration and running time not measured; training cost is estimated by memory counts rather than hardware implementation.
- Theoretical analysis assumes strong convexity and smoothness, and only simplified quantization (parameter-only) is analyzed; experiments are on small ConvNet/MLP models and small image datasets.
Experimental setup
Datasets: FashionMNIST CIFAR10 CIFAR100 CINIC10
Benchmarks: FashionMNIST CIFAR10 CIFAR100 CINIC10
Metrics: test accuracy test loss communication cost (MB/round) training cost (MB/client)
Baselines: FedAvg FedProx ABAvg FedFTG FedGen HeteroFL SplitMix
Builds on: FedAvg FedProx FedGen FedFTG ABAvg
Closest papers by meaning
- FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.91)
- Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.90)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.90)
- (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning (NeurIPS 2024 · similarity 0.90)
- Communication-Efficient Federated Learning with Accelerated Client Gradient (CVPR 2024 · similarity 0.90)
- A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.