Research map / Federated learning and privacy / Communication-efficient FL
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedKSeed, a federated full-parameter tuning method for billion-sized LLMs that transmits only a finite pool of K random seeds and scalar gradients (under 18 KB per round), plus a non-uniform seed sampling variant FedKSeed-Pro.
Problem
Federated full-parameter fine-tuning of billion-sized LLMs on devices is impractical because transmitting model parameters (or replaying unbounded seed histories) incurs prohibitive communication, computation, or memory costs, while PEFT methods may underperform full-parameter tuning.
Approach
Use zeroth-order optimization with two-point gradient estimates where perturbations are drawn from a finite pool of K seeds, so the server only aggregates scalar gradients per seed and clients replay at most K updates to sync; a probability-differentiated seed sampling (based on average scalar gradient magnitude) further reduces the needed K and improves accuracy.
Key results
- FedKSeed-Pro achieves best practical accuracy in most settings, e.g., 23.50 Rouge-L on Natural Instructions with DataJuicer-1.3B vs 22.30 for FedIT.
- Per-round communication is about 17,988 bytes (under 18 KB) for FedKSeed versus 12.76 GB for FedAvg on LLaMA-3B.
- On Dolly-15K (alpha=0.5) with LLaMA-3B, FedKSeed-Pro outperforms the best practical baseline FedIT by 3.06%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full-parameter FedKSeed instead of the strongest practical baseline | DataJuicer-1.3B and LLaMA-3B · Natural Instructions test | Rouge-L: 22.30 -> 22.33; 28.14 -> 29.77 | Helped |
| Use full-parameter FedKSeed-Pro instead of the strongest practical baseline | DataJuicer-1.3B and LLaMA-3B · Dolly-15K (α=0.5) test | Rouge-L: 32.73 -> 33.18; 33.23 -> 36.29 | Helped |
| Use FedKSeed-Pro instead of FedIT | DataJuicer-1.3B and LLaMA-3B · Dolly-15K (α=5.0) test | Rouge-L: 30.97 -> 33.00; 33.68 -> 35.95 | Helped |
| Add non-uniform seed probabilities based on scalar-gradient magnitude (ablation) | FedKSeed · Natural Instructions and Dolly-15K test scenarios | Rouge-L: 22.33 -> 23.50; 29.77 -> 30.19; 32.90 -> 33.18; 35.64 -> 36.29; 33.12 -> 33.00; 35.93 -> 35.95 | Mixed |
| Reduce the candidate seed count below the recommended range (ablation) | FedKSeed and FedKSeed-Pro · Rouge-L evaluation scenarios | Rouge-L | Hurt |
| Increase the candidate seed count beyond the recommended range (ablation) | FedKSeed and FedKSeed-Pro · Rouge-L evaluation scenarios | Rouge-L | No clear effect |
| Use FedKSeed-Pro instead of FedKSeed (ablation) | DataJuicer-1.3B · Federated round communication | communication: 17,988 B -> 9,796 B | Helped |
| Use FedKSeed-Pro instead of FedKSeed (ablation) | LLaMA-3B · Federated round communication | communication: 17,988 B -> 17,988 B | No clear effect |
Surprising findings
- Full-parameter ZOO-based federated methods can match or exceed PEFT baselines in accuracy on held-out tasks while using far less communication.
- FedPrompt collapses on Natural Instructions (6.04 Rouge-L with DataJuicer-1.3B), showing large variance across PEFT methods.
Limitations
- Convergence guarantees rely on analysis borrowed from FedZO and a mean-shift approximation; the seed restriction is argued rather than fully proven.
- Accuracy depends on choosing K within a range (about 1024-4096) and on hyperparameters such as learning rate and epsilon.
- Evaluated only on two instruction datasets with models of 1.3B and 3B parameters.
Experimental setup
Datasets: Natural Instructions v2.8 Dolly-15K
Benchmarks: Natural Instructions (held-out tasks) Dolly-15K
Metrics: Rouge-L communication cost (bytes) peak GPU memory
Baselines: FedPTuning FedPrompt FedIT FedIT-SGD FedAvg FedZO FedMeZO
Builds on: MeZO FedAvg FedZO FedIT
Closest papers by meaning
- Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models (ICML 2025 · similarity 0.93)
- Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.92)
- FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models (ICML 2024 · similarity 0.92)
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.91)
- FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.91)
- Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.