Research map / Federated learning and privacy / Personalized and heterogeneous FL

Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model

AAAI 2025 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes (as a research plan without experiments) a federated system where small LLM clients send task vectors that a large server model aggregates via weighted task-vector addition, with linear projection to map between model sizes.

Problem

Federated learning with heterogeneous clients whose models differ greatly in size and architecture, so standard aggregation such as FedAvg does not apply to integrating small client LLMs into a large server model.

Approach

Clients fine-tune small LLMs (e.g., bloom-560m) and upload task vectors; the server projects these into its larger weight space via learned linear maps and aggregates them with weighted task-vector updates, optionally combined with knowledge distillation, parameter interpolation, and LARS.

Key results

• No experimental results are reported; the paper states expectations only (improved server performance with heterogeneous clients).

Limitations

• No implementation or experiments; the method is a proposal with unspecified details of the projection and aggregation.
• Weighting scheme for alpha_i and the linear projection are described only at a high level.
• Evaluation is planned, not performed.

Experimental setup

Datasets: XL-Sum IN22 FLORES Ethos COPA

Benchmarks: XL-Sum IN22 FLORES Ethos COPA

Metrics: ROUGE BLEU chrF Accuracy communication efficiency overload resource utilization

Baselines: FedAvg FedProx

Builds on: FedAvg FedProx task arithmetic LARS Federated Knowledge Distillation

Closest papers by meaning

• FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.92)
• Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.90)
• FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.90)
• Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneity (AAAI 2024 · similarity 0.89)
• A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.89)
• One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models (ICML 2025 · similarity 0.89)