Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedOne, a federated black-box discrete prompt learning method that activates only one client per round to minimize LLM query cost, with a convergence analysis and experiments on RoBERTa-large and GPT-3.5 Turbo.
Problem
Federated black-box discrete prompt tuning of cloud-hosted LLMs incurs substantial query costs per client, which prior federated BDPL/BBT work ignored; no convergence or query-efficiency analysis existed for federated BDPL.
Approach
Theoretical analysis shows query count grows linearly with the number of activated clients while convergence gains are sublinear, so degrading FedAvg to one active client per round (FedOne) is query-optimal; the client-side learner uses Gumbel-Softmax reparameterized policy gradient with MB-SVRP variance reduction.
Key results
- On RoBERTa-large GLUE, FedOne-GS-BDPL averages 54.90 vs 47.33 for manual prompt and 57.96 for fine-tuning.
- On GPT-3.5 Turbo, FedOne-GS-BDPL averages 64.28 across seven GLUE tasks vs 59.64 for no prompt.
- FedOne uses 100 FL-server queries versus 1000 for 10-client federated training, reducing LLM server queries from 20000 to 2000 for BDPL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gumbel-Softmax BDPL instead of BDPL (ablation) | FedOne-BDPL · GLUE Avg. (GPT-3.5 Turbo) | score: 61.67 -> 64.28 | Helped |
| Use Gumbel-Softmax BDPL instead of BDPL (ablation) | FedOne-BDPL · MNLI (GPT-3.5 Turbo) | score: 13.67 -> 16.00_{1.78} | Helped |
| Use Gumbel-Softmax BDPL instead of BDPL (ablation) | FedOne-BDPL · QQP (GPT-3.5 Turbo) | score: 68.25_{3.05} -> 69.92_{2.85} | Helped |
| Use Gumbel-Softmax BDPL instead of BDPL (ablation) | FedOne-BDPL · SST-2 (GPT-3.5 Turbo) | score: 87.77_{3.82} -> 92.58_{2.71} | Helped |
| Use Gumbel-Softmax BDPL instead of BDPL (ablation) | FedOne-BDPL · MRPC (GPT-3.5 Turbo) | score: 81.69_{0.70} -> 82.93_{0.67} | Helped |
| Use Gumbel-Softmax BDPL instead of BDPL (ablation) | FedOne-BDPL · CoLA (GPT-3.5 Turbo) | score: 32.16_{6.51} -> 36.20_{2.95} | Helped |
| Use Gumbel-Softmax BDPL instead of BDPL (ablation) | FedOne-BDPL · QNLI (GPT-3.5 Turbo) | score: 70.58_{1.35} -> 72.05_{1.64} | Helped |
| Use Gumbel-Softmax BDPL instead of BDPL (ablation) | FedOne-BDPL · RTE (GPT-3.5 Turbo) | score: 77.62_{1.62} -> 80.33_{2.32} | Helped |
| Activate one client per round instead of the federated BDPL configuration | Fed-BDPL · Federated GLUE | LLM server queries: 20000 -> 2000 | Helped |
| Activate one client per round instead of the federated BDPL configuration | Fed-BDPL · Federated GLUE | training time (s): 2407.05 -> 234.92 | Helped |
| Tune a random prompt rather than use no prompt | GPT-3.5 Turbo · GLUE Avg. | score: 59.64 -> 54.41 | Hurt |
| Tune the discrete prompt with FedOne-GS-BDPL rather than use no prompt | GPT-3.5 Turbo · GLUE Avg. | score: 59.64 -> 64.28 | Helped |
Surprising findings
- Fine-tuning and manual prompting are not uniformly best; black-box BDPL variants match or slightly exceed white-box prompt tuning on several tasks.
- Adding an untrained random prompt to GPT-3.5 lowers performance relative to no prompt.
Limitations
- Fewer activated clients per round yields slower convergence per round.
- Real-world GPT-3.5 experiments rely on top-20 log-probabilities, requiring a template question to get label probabilities.
Experimental setup
Datasets: GLUE MNIST MNLI QQP SST-2 MRPC CoLA QNLI RTE
Benchmarks: GLUE MNIST
Metrics: accuracy query count computation time communication cost
Baselines: Manual Prompt In-Context Learning Fine-tuning Prompt Tuning P-Tuning v2 BBT BDPL
Builds on: BDPL FedAvg Gumbel-Softmax BBT P-Tuning v2
Closest papers by meaning
- FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models (ICML 2024 · similarity 0.90)
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.88)
- FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.88)
- Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data (NeurIPS 2024 · similarity 0.87)
- FLiP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning (NeurIPS 2025 · similarity 0.86)
- DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.