Research map / Large language models / Instruction and fine-tuning
FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the FlowerTune LLM Leaderboard, a community-driven benchmark for federated fine-tuning of 26 pre-trained LLMs (135M–14B) across general NLP, finance, medical and coding domains, with FedAvg/FedProx/FedAvgM/FlexLoRA and LoRA/DoRA comparisons.
Problem
Pre-trained LLMs' compatibility and performance under federated fine-tuning across domains (and under FL resource constraints such as communication and memory) remain largely unexplored, with no standardized benchmark.
Approach
A unified federated benchmark pairing four domain-specific instruction datasets partitioned into clients with domain evaluation suites, used to systematically compare many base models, FL aggregation strategies and PEFT methods while logging communication and VRAM costs.
Key results
- Qwen2.5-7B-Instruct achieves the highest general NLP average (64.50%), and Gemma2-9B-Instruct leads finance (84.21%) and medical (62.25%) averages.
- Gemma2-9B-Instruct reaches 53.29% average pass@1 on coding, with Llama3.1-8B-Instruct close at 53.16%.
- Aggregation strategies (FedAvg, FedProx, FedAvgM, FlexLoRA) yield only marginal differences on Qwen2.5-7B, and LoRA vs DoRA differ little; all memory footprints stay below 80 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select Qwen2.5-7B-Instruct as the base model rather than the next-best listed model | Federated fine-tuning on General NLP · General NLP (MMLU) | average accuracy (%): 63.48 -> 64.50 | Helped |
| Select Gemma2-9B-Instruct rather than Mistral-7B-Instruct-v0.3 | Federated fine-tuning on Finance · Finance | average accuracy (%): 69.60 -> 84.21 | Helped |
| Select Gemma2-9B-Instruct rather than Qwen2.5-7B-Instruct | Federated fine-tuning on Medical · Medical | average accuracy (%): 57.88 -> 62.25 | Helped |
| Select Gemma2-9B-Instruct rather than Llama3.1-8B-Instruct | Federated fine-tuning on Coding · Coding | average Pass@1 (%): 53.16 -> 53.29 | Helped |
| Use FedProx instead of FedAvg (ablation) | Qwen2.5-7B federated tuning · General NLP | average accuracy (%): 42.79 -> 42.92 | Helped |
| Use FedAvg instead of FedProx (ablation) | Qwen2.5-7B federated tuning · Medical | average accuracy (%): 23.41 -> 25.02 | Helped |
| Use FedAvgM instead of FedAvg (ablation) | Qwen2.5-7B federated tuning · Finance | average accuracy (%): 66.12 -> 68.31 | Helped |
| Use FedAvgM instead of FedAvg (ablation) | Qwen2.5-7B federated tuning · Coding | average Pass@1 (%): 45.25 -> 46.77 | Helped |
| Replace DoRA with LoRA (ablation) | Qwen2.5-7B federated tuning · General NLP | average accuracy (%): 40.18 -> 42.79 | Hurt |
| Replace DoRA with LoRA (ablation) | Qwen2.5-7B federated tuning · Medical | average accuracy (%): 25.39 -> 25.02 | No clear effect |
| Replace DoRA with LoRA (ablation) | Qwen2.5-7B federated tuning · Finance | average accuracy (%): 66.12 -> 68.01 | Helped |
| Replace DoRA with LoRA (ablation) | Qwen2.5-7B federated tuning · Coding | average Pass@1 (%): 45.25 -> 47.87 | Helped |
Surprising findings
- Smaller models such as SmolLM2-135M-Instruct and Qwen2.5-1.5B-Instruct are competitive with larger models on some challenges, e.g. finance classification.
- Non-instruct base models generally underperform their instruct counterparts after federated fine-tuning.
Limitations
- Focuses on cross-silo FL settings; cross-device and more heterogeneous scenarios not covered.
- No per-model hyper-parameter tuning; uniform configuration may understate some models.
- Aggregation algorithms show only marginal differences, and results depend on the chosen evaluation datasets.
Experimental setup
Datasets: alpaca-gpt4 fingpt-sentiment-train medical-flashcards code-alpaca-20k MMLU FPB FIQA TFNS PubMedQA MedMCQA MedQA CareQA
Benchmarks: MMLU FPB FIQA TFNS PubMedQA MedMCQA MedQA CareQA MBPP HumanEval MultiPL-E (JS) MultiPL-E (C++)
Metrics: accuracy pass@1 communication cost (GB) VRAM usage (GB)
Baselines: FedProx FedAvgM FlexLoRA LoRA
Builds on: FedAvg FedProx FlexLoRA LoRA DoRA
Closest papers by meaning
- FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.92)
- FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.91)
- FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models (ICML 2024 · similarity 0.91)
- Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources (NeurIPS 2024 · similarity 0.91)
- Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.