Research map / Federated learning and privacy / Personalized and heterogeneous FL
Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pFedSeq, which personalizes LoRA adapters for a frozen ViT-B/16 in federated learning by training a server-side Mamba-based sequential learner over the history of client adapter updates.
Problem
Existing personalized federated learning (PFL) methods build personalized models only from clients' latest updates, discarding or underusing previous update trajectories, which yields suboptimal personalized adapters for federated adapter tuning of foundation models.
Approach
A server-side hypernetwork-style sequential learner (2-layer Selective SSM / Mamba) takes the sequence of past client adapter updates, models cross-client and cross-step relations, and outputs personalized calibrations added to the FedAvg global adapter; it is trained using client updates as gradient proxies, without accessing client data.
Key results
- pFedSeq reaches 95.30% on CIFAR-100, 94.30% on Tiny-ImageNet, 84.63% on DomainNet and 45.25% on Omniglot, the best of 12 compared methods, with margins of 0.84%, 0.6%, 2.21%, and 2.27% over the second-best (pFedHN or PeFLL).
- Ablations show removing global aggregation causes the largest drop (0.89% on CIFAR-100, 1.96% on DomainNet); replacing the Mamba learner with MLP or LSTM lowers DomainNet accuracy by 3.38% and 1.86%.
- On Omniglot, accuracy increases with the maximum sequence length L, with L=15 giving a 1.17% gain over L=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pFedSeq instead of pFedHN | federated adapter tuning · CIFAR-100 | accuracy: 94.46 -> 95.30 | Helped |
| Use pFedSeq instead of PeFLL | federated adapter tuning · Tiny-ImageNet | accuracy: 93.70 -> 94.30 | Helped |
| Use pFedSeq instead of PeFLL | federated adapter tuning · DomainNet | accuracy: 82.42 -> 84.63 | Helped |
| Use pFedSeq instead of PeFLL | federated adapter tuning · Omniglot | accuracy: 42.98 -> 45.25 | Helped |
| add global aggregation (ablation) | pFedSeq · CIFAR-100 | accuracy: 94.41 -> 95.30 | Helped |
| add global aggregation (ablation) | pFedSeq · DomainNet | accuracy: 82.67 -> 84.63 | Helped |
| add cross-step modeling of previous updates (ablation) | pFedSeq · CIFAR-100 | accuracy: 94.66 -> 95.30 | Helped |
| add cross-step modeling of previous updates (ablation) | pFedSeq · DomainNet | accuracy: 83.10 -> 84.63 | Helped |
| add cross-client modeling (ablation) | pFedSeq · CIFAR-100 | accuracy: 94.40 -> 95.30 | Helped |
| add cross-client modeling (ablation) | pFedSeq · DomainNet | accuracy: 83.85 -> 84.63 | Helped |
| replace MLP with Selective SSM learner (ablation) | pFedSeq · CIFAR-100 | accuracy: 94.32 -> 95.30 | Helped |
| replace MLP with Selective SSM learner (ablation) | pFedSeq · DomainNet | accuracy: 81.25 -> 84.63 | Helped |
Surprising findings
- Even with L=1 (no cross-step history), pFedSeq outperforms PeFLL on Omniglot, indicating that the architecture and global aggregation contribute beyond cross-step modeling.
- Removing global aggregation leaves the variant competitive with the strongest baselines, yet it still performs worse than the full model.
Limitations
- Experiments use a single backbone (ViT-B/16 with LoRA) and image classification only.
- Requires storing a sequence of past client updates at the server, with a cap on sequence length L and a warm-up period W of 10 rounds during which personalized adapters are not used.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet DomainNet Omniglot ImageNet21k CIFAR-10
Benchmarks: CIFAR-100 Tiny-ImageNet DomainNet Omniglot
Metrics: accuracy (mean over clients, 3 seeds) time per communication round
Baselines: Local FedAvg Per-FedAvg pFedMe APPLE FedALA FedRep Ditto PerAda pFedHN pFedLA PeFLL
Builds on: FedAvg pFedHN PeFLL Mamba LoRA
Closest papers by meaning
- FedAS: Bridging Inconsistency in Personalized Federated Learning (CVPR 2024 · similarity 0.90)
- pFedClub: Controllable Heterogeneous Model Aggregation for Personalized Federated Learning (NeurIPS 2024 · similarity 0.88)
- PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees (CVPR 2024 · similarity 0.88)
- Dual-Personalizing Adapter for Federated Foundation Models (NeurIPS 2024 · similarity 0.88)
- PeFLL: Personalized Federated Learning by Learning to Learn (ICLR 2024 · similarity 0.87)
- Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guarantees (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.