Research map / Federated learning and privacy / Personalized and heterogeneous FL

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models

ICML 2025 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Non-IID federated learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedICU, a federated LoRA fine-tuning framework for LLMs that splits client updates into consensus (magnitude) and divergence (direction) parts and uploads only importance-selected parameters to reduce forgetting under non-IID instruction data.

Problem

Federated supervised fine-tuning of LLMs on heterogeneous (non-IID) client instruction data tends to overfit to local distributions and cause catastrophic forgetting of general capabilities, while uploading all LoRA parameters is communication-costly.

Approach

Decompose each client's LoRA update into column-wise L2-norm (consensus) and normalized direction (divergence); aggregate consensus by averaging and divergence by cosine-similarity-weighted softmax, and perform momentum-based importance-aware masked uploads that keep only parameters whose specialization importance exceeds their generalization importance.

Key results

• FedICU reaches MT-Final 4.13 vs 3.80 for FedAvg on MT-Bench with Llama-2-7b, and achieves the best average rank of 1.8 across benchmarks.
• Ablation on MT-Bench: adding only Consensus-Divergence Splitting raises MT-Final from 3.80 to 4.06; adding only Importance-Aware Updating gives 3.90; combined gives 4.13.
• Importance-aware mask reduces uploaded parameters; overlap of important parameters across clients drops from 0.24 to 0.08 over rounds 5 to 65 (Table 3).

Measured effects reported in the paper

Surprising findings

• Importance-aware updating alone slightly lowers MT-1 (4.55 vs 4.58 baseline FedAvg) while improving MT-2, indicating trade-offs between first-turn and contextual scores.
• Several FedICU gains on HumanEval, MMLU and GSM8K are small or tied with baselines (e.g., HumanEval 0.35 vs 0.35 for FedProx-like baselines; GSM8K 0.10 vs 0.10 for some methods).

Limitations

• Relies on LoRA, which may not match full-parameter fine-tuning performance (acknowledged by authors).
• Evaluated only on Llama-2-7b with four clients and one dataset per client; generality across model sizes and client counts is limited.
• Communication savings are stated only theoretically in big-O form; no measured communication numbers reported.

Experimental setup

Datasets: Alpaca (Taori et al., 2023) Code dataset (Chaudhary, 2023) MathInstruct/GSM8K-style math dataset (Xiang Yue, 2023) FinGPT sentiment dataset (Yang et al., 2023b)

Benchmarks: MT-Bench HumanEval MMLU GSM8K

Metrics: MT-Bench first-turn score MT-Bench second-turn score MT-Bench final score HumanEval score MMLU score GSM8K score average rank

Baselines: FedAvg FedProx FedAvgM Scaffold FedAdam FedYogi Base model without SFT

Builds on: OpenFedLLM FedAvg LoRA FedProx Scaffold

Closest papers by meaning

• FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.91)
• FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.89)
• Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.89)
• FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models (NeurIPS 2024 · similarity 0.88)
• FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.88)
• Thinking Forward: Memory-Efficient Federated Finetuning of Language Models (NeurIPS 2024 · similarity 0.88)