Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedBPT, a federated black-box prompt tuning framework where clients optimize shared low-dimensional prompt parameters with CMA-ES using only forward inference, plus server-level CMA-ES aggregation and perturbation-based regularization against local overfitting.
Problem
Federated fine-tuning of pre-trained language models is hampered by clients lacking parameter access, high on-device compute and memory costs, and heavy communication overhead.
Approach
Clients tune a prompt via gradient-free CMA-ES (building on BBT) using only inference, exchange only low-dimensional CMA-ES statistics, aggregate them with a server-level CMA-ES using a corrected step size, and regularize local search by dividing loss by the loss on randomly token-perturbed inputs to prevent label-skew overfitting.
Key results
- On RoBERTa under non-IID SST-2, FedBPT reaches 86.47% accuracy, 0.92 points above FedPrompt (85.55%) and 0.69 below FedP-tuning (87.16%) with only 500 trainable parameters.
- FedBPT reduces per-round communication for one device from 120MB (FedP-tuning) to 4KB, and client memory from 7.2 GB (FedAvg) to 1.8 GB on SST-2.
- With Llama 2 (7B), FedBPT improves accuracy by more than 12% on SST-2 non-IID over manual prompts while using 500 trainable parameters versus 235M for FedP-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FedBPT instead of FedAvg-BBT | RoBERTa · SST-2, AG's News, and Yelp non-IID | accuracy (%): 2.3%, 4.57%, and 1.08% | Helped |
| use FedBPT instead of gradient-based baselines | RoBERTa · SST-2, AG's News, and Yelp IID and non-IID | accuracy (%): 0.92%; 0.69% | Mixed |
| use FedBPT instead of gradient-based methods | RoBERTa · SST-2 | memory footprint: more than $3\times$ | Helped |
| use FedBPT instead of manual prompts | Llama 2 · SST-2 non-IID | accuracy: more than 12% | Helped |
| use FedBPT instead of manual prompts | Llama 2 · AG's News non-IID | accuracy: 11% | Helped |
| use FedBPT instead of manual prompts | Llama 2 · Yelp non-IID | accuracy: 13% | Helped |
| use FedBPT instead of FedPrompt | Llama 2 · AG's News IID | accuracy | Helped |
| replace direct FedAvg aggregation with server-level CMA-ES aggregation (ablation) | FedBPT · language understanding datasets | accuracy | Helped |
| apply random token replacement rather than vanilla BBT local training (r_p=0) (ablation) | FedBPT · SST-2 non-IID | accuracy (%): 84.86 -> 85.21; 84.86 -> 86.03; 84.86 -> 86.47; 84.86 -> 86.12 | Helped |
| set local binary mask rate r_p to 0.2, 0.4, 0.6, or 0.8 instead of 0 (ablation) | FedBPT · AG's News non-IID | accuracy (%): 78.28 -> 80.92; 78.28 -> 81.03; 78.28 -> 80.75; 78.28 -> 80.83 | Helped |
| vary local population size λ_k from 5 to 20 (ablation) | FedBPT · SST-2 and AG's News | accuracy | No clear effect |
Surprising findings
- Directly applying FedAvg to fine-tune the full RoBERTa model underperforms the zero-shot manual prompt baseline on AG's News and Yelp, indicating full fine-tuning is infeasible with few labeled samples.
- Simply combining BBT with FedAvg (FedAvg-BBT) yields only limited gains over manual prompts, showing naive aggregation fails.
Limitations
- Gradient-based methods still outperform FedBPT in several settings, notably with RoBERTa on AG's News and Yelp under some splits.
- Experiments are limited to few-shot settings (40 samples per class) on three classification datasets with ten clients.
- Server-level step-size correction relies on assumptions (neutral selection, approximately constant local covariance) and theoretical derivations are partly approximate.
Experimental setup
Datasets: SST-2 Yelp polarity AG's News
Benchmarks: SST-2 AG's News Yelp polarity
Metrics: accuracy trainable parameters communication cost memory footprint
Baselines: FedPrompt FedP-tuning FedAvg Manual prompt In-context learning FedAvg-BBT
Builds on: Black-box Tuning (BBT) FedAvg FedPrompt P-tuning CMA-ES
Closest papers by meaning
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.92)
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.91)
- Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models (ICML 2025 · similarity 0.91)
- FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learning (ICML 2025 · similarity 0.90)
- Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.90)
- FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.