Research map / Federated learning and privacy / Communication-efficient FL
Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ferret, a federated full-parameter LLM tuning method that combines first-order local updates with random-basis projection and block-wise reconstruction from shared seeds, cutting per-round communication to O(K) scalars.
Problem
Federated full-parameter fine-tuning of LLMs suffers from either high communication cost (first-order FL transmits O(d) parameters) or poor computational efficiency and slow convergence (zeroth-order FL with shared randomness, e.g., FedKSeed).
Approach
Clients run first-order local updates, project the update onto K random bases generated from a shared seed, and the server reconstructs full updates from seeds and projected coefficients; an approximate inversion-free reconstruction and block-wise random bases make this scalable and unbiased.
Key results
- On Natural Instructions with LLaMA-3B, Ferret achieves 30.03 Rouge-L vs 32.11 for FedAvg and 29.77 for FedKSeed, with ~6.0x lower overall per-round compute than FedKSeed and ~10^6x lower communication than FedAvg.
- Ferret converges in about 2 communication rounds, comparable to FedAvg, versus 40 rounds for FedKSeed on Natural Instructions.
- On GSM8K with LLaMA2-7B, Ferret reaches 36.10 vs FedKSeed 28.26 but below FedAvg 38.30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ferret instead of FedAvg | Federated full-parameter tuning · Natural Instructions, DataJuicer-1.3B | Rouge-L (%): 23.95 ± 2.76 -> 24.99 ± 0.99 | Helped |
| Use Ferret instead of FedAvg | Federated full-parameter tuning · Natural Instructions, LLaMA-3B | Rouge-L (%): 32.11 ± 0.70 -> 30.03 ± 0.99 | Hurt |
| Use Ferret instead of FedKSeed | Federated full-parameter tuning · Dolly-15K, DataJuicer-1.3B | Rouge-L (%): 30.91 ± 0.29 -> 30.63 ± 0.84 | Hurt |
| Use Ferret instead of FedKSeed | Federated full-parameter tuning · Dolly-15K, LLaMA-3B | Rouge-L (%): 34.56 ± 0.28 -> 34.57 ± 0.57 | No clear effect |
| Use Ferret instead of FedAvg | Federated full-parameter tuning · CodeAlpaca, LLaMA2-7B | Rouge-L (%): 15.41 ± 0.43 -> 12.10 ± 0.47 | Hurt |
| Use Ferret instead of FedAvg | Federated full-parameter tuning · CodeAlpaca, LLaMA2-13B | Rouge-L (%): 14.68 ± 0.26 -> 11.84 ± 0.91 | Hurt |
| Use Ferret instead of FedAvg | Federated full-parameter tuning · GSM8K, LLaMA2-7B | Rouge-L (%): 38.30 ± 0.40 -> 36.10 ± 1.18 | Hurt |
| Use Ferret instead of FedAvg | Federated full-parameter tuning · GSM8K, LLaMA2-13B | Rouge-L (%): 39.82 ± 0.17 -> 34.50 ± 1.42 | Hurt |
| Replace FedKSeed with Ferret | Federated tuning on LLaMA-3B · Per-round overall computational cost | seconds: 180.7 -> 30.3 (6.0×) | Helped |
| Use shared randomness and projected coordinates instead of full-model communication | Federated tuning on LLaMA-3B · Per-round communication | number of parameters communicated: 6.0×10^9 -> 7.8×10^3 (10^6×) | Helped |
| Increase the number of random bases K (ablation) | Ferret · Natural Instructions, DataJuicer-1.3B | convergence | Helped |
| Increase K beyond 2K₀ (ablation) | Ferret · Natural Instructions, DataJuicer-1.3B | generalization performance | Hurt |
Surprising findings
- Ferret slightly underperforms FedAvg on complex tasks such as CodeAlpaca and GSM8K, which the authors attribute to reconstruction error.
Limitations
- Privacy analysis is not provided; the authors defer in-depth privacy analysis to future work.
- Slightly lower accuracy than FedAvg on CodeAlpaca and GSM8K due to reconstruction error.
- Experiments use small base models and limited hardware resources (DataJuicer-1.3B, LLaMA-3B).
Experimental setup
Datasets: Natural Instructions Dolly-15K CodeAlpaca GSM8K
Benchmarks: Natural Instructions Dolly-15K CodeAlpaca GSM8K
Metrics: Rouge-L computational cost per round (sec) communication cost (# params) communication rounds to converge peak GPU memory
Baselines: FedPTuning FedPrompt FedIT FedIT-SGD FedZO FedKSeed FedAvg FedMeZO
Builds on: FedKSeed FedAvg MeZO Adam
Closest papers by meaning
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.93)
- Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.92)
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.91)
- FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models (ICML 2024 · similarity 0.91)
- Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages (ICLR 2024 · similarity 0.89)
- Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.