Research map / Federated learning and privacy / Communication-efficient FL

FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates

ICML 2025 · Federated learning and privacy › Communication-efficient FL · approach: Federated optimization algorithms

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedSMU, a federated learning method that uploads only the sign of client model updates (1-bit) and splits the Lion optimizer between client and server, improving communication efficiency and generalization under data heterogeneity.

Problem

Federated learning suffers from high communication overhead and data heterogeneity, and existing compression methods and optimization-based methods typically address only one of these, often at the cost of generalization or extra communication.

Approach

Symbolize client model updates (1-bit sign after local momentum tracking) so that each client contributes equally in magnitude, and split the centralized Lion optimizer so only the update signs are communicated, with momentum and weight decay handled locally and globally.

Key results

• On CIFAR-100 with LeNet (Dir 0.25, 100 clients, 10% participation), FedSMU reaches 51.87-54.2% top-1 accuracy across settings versus 41.44% for FedAvg and 45.09% for FedLion in the low-participation Dir(0.25) setting.
• With ResNet18 on Tiny-ImageNet (Dir 0.25), FedSMU attains 33.53% vs 29.79% for FedAvg and 33.35% for SCAFFOLD.
• With ViT-S on CIFAR-100 (Dir 0.25), FedSMU reaches 53.24% vs 41.72% for FedAvg and 51.79% for FedEF-HS; on CIFAR-10 with ResNet18 it trails SCAFFOLD (83.54% vs 85.62%).

Measured effects reported in the paper

Surprising findings

• Higher-bit (3-bit, 8-bit) symbol-like quantization did not improve convergence or generalization over 1-bit in the FedSMU design.
• FedSMU does not reduce the number of communication rounds; it sometimes needs more rounds than baselines on CIFAR-10/100 despite lower bit cost.

Limitations

• Fixed-precision 1-bit symbol quantization may not be optimal for adaptive scenarios.
• Server-to-client broadcast of the full global model is not compressed, and sign noise slows convergence in rounds.
• Only SGD is used for local updates; adaptive local optimizers and large language models are not evaluated.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet Shakespeare

Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet Shakespeare

Metrics: top-1 test accuracy communication bits communication rounds generalization at matched training accuracy

Baselines: FedAvg SCAFFOLD SCALLION FedEF-HS FedEF-TopK FedEF-Sign FedLion

Builds on: Lion signSGD FedAvg SCAFFOLD FedLion

Closest papers by meaning

• Communication-Efficient Federated Learning with Accelerated Client Gradient (CVPR 2024 · similarity 0.90)
• FedASMU: Efficient Asynchronous Federated Learning with Dynamic Staleness-Aware Model Update (AAAI 2024 · similarity 0.90)
• FedImpro: Measuring and Improving Client Update in Federated Learning (ICLR 2024 · similarity 0.90)
• Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.90)
• FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.90)
• Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.90)