Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FedLLM-Bench, a set of user-partitioned real-world federated datasets (Fed-Aya, Fed-ChatbotIT, Fed-WildChat, Fed-ChatbotPA) with 8 FL training methods and 6 evaluation metrics for benchmarking federated LLM instruction tuning and preference alignment.
Problem
There is no realistic benchmark or dataset for federated learning of LLMs (FedLLM); prior work relies on artificially partitioned centralized datasets, leading to unrealistic evaluation and unfair comparisons.
Approach
Build four federated datasets naturally split by real user IDs (annotator, IP address, or user ID) from Aya, Chatbot Arena Conversations, and WildChat, spanning instruction tuning and preference alignment, and integrate them with classical FL baselines and LLM evaluation metrics into one codebase.
Key results
- On Fed-Aya (Ref-GPT4), FedAvg averages 4.90 and FedProx 4.92 versus 4.18-4.66 for single-language local training; no FL method dominates all languages.
- On Fed-ChatbotIT, FedAdagrad achieves the best average (5.48) and ties for best HumanEval (15.85) with FedYogi; all FL methods beat local training on open-ended metrics.
- On Fed-ChatbotPA, FL methods raise AdvBench safety from 11.0 (local) to as high as 16.2 (FedAvg), and FedAvgM reaches the best MT-Bench-1 of 4.71.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use federated training instead of local training | Fed-Aya · Fed-Aya, average across languages | Ref-GPT4: 4.18 -> 4.97 | Helped |
| Use federated training instead of local training | Fed-ChatbotIT · Fed-ChatbotIT | Average: 5.00 -> 5.51 | Helped |
| Use federated training instead of local training | Fed-WildChat · Fed-WildChat, single-turn | MT-1: 4.15 -> 4.86 | Helped |
| Use FedAvg for federated training | Fed-WildChat · Fed-WildChat, multi-turn | MT-Bench: 3.27 -> 3.99 | Helped |
| Use federated preference alignment instead of local training | Fed-ChatbotPA · Fed-ChatbotPA | Average: 5.37 -> 5.79 | Helped |
| Collaborate with same-language clients rather than train locally | Fed-Aya · Fed-Aya, average across languages | Ref-GPT4: 4.69 -> 5.36 | Helped |
| Collaborate with similar-language clients instead of same-language clients | Fed-Aya · Fed-Aya, across languages | Ref-GPT4 | Mixed |
| Add user-level differential privacy with FedDP-1e^-3 | Fed-WildChat FedAvg · Fed-WildChat, single-turn | MT-Bench: 4.6875 -> 4.6750 | No clear effect |
| Increase differential privacy noise to FedDP-1 | Fed-WildChat FedAvg · Fed-WildChat, single-turn | MT-Bench: 4.6875 -> 1.6875 | Hurt |
Surprising findings
- On multi-turn Fed-WildChat, FedAvg consistently outperforms the other FL optimizers, suggesting advanced server-side optimizers do not help in this LLM setting.
- Collaborating with similar languages (FedSimLang) helped only some languages (en, pt, zh) and was worse than same-language collaboration on the others.
Limitations
- Only Llama2-7B and Alpaca-7B are explored as base models, limiting generality across model series and sizes.
- Safety alignment is not comprehensively covered, and the datasets may contain unsafe or toxic interactions.
- Evaluation relies heavily on GPT-4 judging with small in-domain test sets (e.g., 50 samples).
Experimental setup
Datasets: Aya Chatbot-Arena-Conversations WildChat Fed-Aya Fed-ChatbotIT Fed-WildChat Fed-ChatbotPA
Benchmarks: MT-Bench Vicuna bench Ref-GPT4 (in-domain) AdvBench MMLU HumanEval
Metrics: MT-Bench score Vicuna bench score Ref-GPT4 score AdvBench safety rate MMLU accuracy HumanEval pass rate
Baselines: FedAvg FedProx SCAFFOLD FedAvgM FedAdagrad FedYogi FedAdam Local Training
Builds on: FedAvg FedProx SCAFFOLD FedAvgM FedAdam DPO LoRA
Closest papers by meaning
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.92)
- Thinking Forward: Memory-Efficient Federated Finetuning of Language Models (NeurIPS 2024 · similarity 0.89)
- FLHetBench: Benchmarking Device and State Heterogeneity in Federated Learning (CVPR 2024 · similarity 0.88)
- Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models (ICML 2025 · similarity 0.88)
- Benchmarking Algorithms for Federated Domain Generalization (ICLR 2024 · similarity 0.88)
- FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.