Research map / Federated learning and privacy / Communication-efficient FL
Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedMPQ, a federated learning method that assigns per-layer mixed-precision bit-widths to heterogeneous clients via group-Lasso bit-sparsity training and a server-side pruning-growing procedure, outperforming fixed-precision AQFL on CIFAR10/100 and Tiny-ImageNet.
Problem
Federated learning clients have heterogeneous memory and compute budgets, yet existing quantization-aware FL methods use fixed-precision quantization, and mixed-precision quantization methods require full-precision training that violates device constraints.
Approach
Train quantized local models with a group Lasso penalty on binary bit-planes so that MSBs of less sensitive layers become sparse and can be pruned, then have the server aggregate per-layer bit-widths and use a greedy pruning-growing step to match each client's average bit-width budget.
Key results
- On CIFAR10 with alpha=0.1/0.5/1, FedMPQ reaches 49.1/67.1/69.3% test accuracy versus 44.3/58.0/62.1% for AQFL, up to 9.1% higher.
- FedMPQ nearly matches the 8-bit FPQ8 baseline on CIFAR10/100 at alpha=0.5 and 1 (e.g., 67.1 vs 68.4 on CIFAR10 at alpha=0.5).
- Ablation on CIFAR10 shows the pruning-growing step (Alg. 2) combined with MSB pruning and group Lasso lifts accuracy from 56.2% to 67.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mixed-precision instead of fixed-precision quantization | FedMPQ vs AQFL · CIFAR10, CIFAR100, and Tiny-ImageNet | test accuracy: FedMPQ: 49.1, 67.1, 69.3; 31.7, 41.1, 43.6; 20.3, 27.0, 28.2. AQFL: 44.3, 58.0, 62.1; 23.8, 32.9, 36.1; 17.1, 23.5, 25.3 | Helped |
| use FedMPQ instead of FPQ8 | FedMPQ · CIFAR10 and CIFAR100, α=0.5 and 1 | test accuracy: CIFAR10: 67.1 vs 68.4, 69.3 vs 70.8; CIFAR100: 41.1 vs 41.3, 43.6 vs 42.3 | Mixed |
| increase the number of clients from 10 to 20 or 40 | FedMPQ · CIFAR10 and CIFAR100, α=0.5 | test accuracy: CIFAR10: 67.1, 56.8, 45.4; CIFAR100: 41.1, 26.1, 19.9 | Hurt |
| use 1 local epoch rather than more local epochs | quantization-aware training methods, including FedMPQ · CIFAR10 and CIFAR100 | test accuracy | Hurt |
| increase the MSB-pruning threshold ϵ (ablation) | FedMPQ · CIFAR10, α=0.5 | test accuracy: ϵ=0.01: 66.4, 68.0, 67.1, 64.7, 64.3; ϵ=0.02: 67.1, 70.0, 65.5, 64.7, 64.4; ϵ=0.03: 65.9, 67.1, 65.5, 64.7, 64.4 | Hurt |
| increase the group Lasso weight to λ=0.1 (ablation) | FedMPQ · CIFAR10, α=0.5 | test accuracy: λ=0.1: 64.1, 63.4, 65.9, 63.9, 64.8; λ=0.01: 66.4, 68.0, 67.1, 64.7, 64.3 | Hurt |
| add group Lasso regularization (ablation) | FedMPQ without MSBs pruning or Algorithm 2 · CIFAR10 and CIFAR100, α=0.5 | test accuracy: 58.0 -> 56.8; 32.9 -> 32.6 | Hurt |
| add MSBs pruning without sparsity-promoting training or Algorithm 2 (ablation) | FedMPQ · CIFAR10 and CIFAR100, α=0.5 | test accuracy: 58.0 -> 55.1; 32.9 -> 28.7 | Hurt |
| combine MSBs pruning with group Lasso, without Algorithm 2 (ablation) | FedMPQ · CIFAR10 and CIFAR100, α=0.5 | test accuracy: 58.0 -> 56.2; 32.9 -> 31.8 | Hurt |
| add Algorithm 2 to MSBs pruning without group Lasso (ablation) | FedMPQ · CIFAR10 and CIFAR100, α=0.5 | test accuracy: 55.1 -> 65.4; 28.7 -> 40.1 | Helped |
| add Algorithm 2 to MSBs pruning with group Lasso (ablation) | FedMPQ · CIFAR10 and CIFAR100, α=0.5 | test accuracy: 56.2 -> 67.1; 31.8 -> 41.1 | Helped |
| add group Lasso to MSBs pruning with Algorithm 2 (ablation) | FedMPQ · CIFAR10 and CIFAR100, α=0.5 | test accuracy: 65.4 -> 67.1; 40.1 -> 41.1 | Helped |
Surprising findings
- Group Lasso regularization lets clients learn highly sparse binary weight representations so that bit-pruning causes no noticeable degradation.
Limitations
- Activations are fixed at 4 bits and only weight precision is mixed.
- Performance gap to FP32 and FPQ8 widens with more clients and extremely heterogeneous data (e.g., 40 clients on CIFAR100: 19.9 vs 31.9 for FPQ8).
- Hyperparameters epsilon and lambda are sensitive; large values degrade accuracy.
Experimental setup
Datasets: CIFAR10 CIFAR100 Tiny-ImageNet
Benchmarks: CIFAR10 CIFAR100 Tiny-ImageNet
Metrics: test accuracy
Baselines: FP32 FedAvg FedPAQ UVeQFed FPQ8 AQFL
Builds on: AQFL BSQ WAGE FedAvg FedPAQ
Closest papers by meaning
- Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneity (AAAI 2024 · similarity 0.90)
- LBI-FL: Low-Bit Integerized Federated Learning with Temporally Dynamic Bit-Width Allocation (ICML 2025 · similarity 0.89)
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.88)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.87)
- Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.86)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.