Research map / Federated learning and privacy / Personalized and heterogeneous FL

Dual-Personalizing Adapter for Federated Foundation Models

NeurIPS 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Sparse and collaboration-graph personalization · 13 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedDPA, a federated dual-adapter method (global LoRA shared via FedAvg plus a client-local LoRA) with instance-wise similarity-based weighting, to achieve personalization and test-time personalization for federated LLMs.

Problem

Federated foundation models are personalized to client tasks, but existing methods ignore test-time distribution shifts (unseen tasks/distributions at inference), and conventional test-time FL methods are ill-suited to LLMs that require training all parameters.

Approach

Each client keeps a frozen LLM with a globally aggregated LoRA adapter (for generic, test-time knowledge) and a local LoRA adapter (for personalization); at inference, the two are mixed per instance using cosine similarity between last-token LLM embeddings of the test input and sampled local training instances.

Key results

• On federated dataset 1, FedDPA-T reaches 80.22 personalization average vs 79.43 for FedLoRA and 78.60 for centralized; FedDPA-F reaches 77.12 test-time personalization vs 75.63 for FedLoRA.
• On federated dataset 2, FedDPA-F reaches 71.64 test-time personalization vs 69.86 for FedLoRA; FedDPA-T reaches 75.33 personalization vs 72.74 for FedLoRA.
• Scaling to LLaMA-13B raises FedDPA-F test-time personalization on dataset 1 from 77.12 to 80.55.

Measured effects reported in the paper

Surprising findings

• Centralized and global models achieve higher average performance across all tasks but underperform on specific local tasks, illustrating the personalization trade-off.
• Without the instance-wise weighting, FedDPA-T's test-time personalization drops sharply (e.g., 60.06 vs 75.47 on dataset 1), showing the mechanism's importance for the training-based local adapter.

Limitations

• Evaluation uses only 8 clients (one per task) on small downsampled datasets (300 train / 200 test per task) with a single metric, ROUGE-1.
• Test-time shift is simulated by mixing tasks other clients train on, rather than real-world distribution shifts.
• Hyperparameters (alpha, lambda) are tuned per dataset and the FedDPA-T weighting is sensitive to them.

Experimental setup

Datasets: Flan (GLUE QQP, SNLI, WebNLG, fix_punct, CoLA, WiC, definite pronoun resolution, TREC; paws_wiki, hellaswag, qnli, word_

Benchmarks: Federated Dataset 1 Federated Dataset 2

Metrics: ROUGE-1 accuracy

Baselines: Centralized Local-finetuned FedIT FedLoRA FedProx

Builds on: LoRA FedAvg FedProx FedIT

Closest papers by meaning

• Adaptive Test-Time Personalization for Federated Learning (NeurIPS 2023 · similarity 0.90)
• Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning (AAAI 2025 · similarity 0.88)
• FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.88)
• PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees (CVPR 2024 · similarity 0.87)
• FedAS: Bridging Inconsistency in Personalized Federated Learning (CVPR 2024 · similarity 0.87)
• Adaptive Latent-Space Constraints in Personalized Federated Learning (NeurIPS 2025 · similarity 0.87)