Research map / Large language models / Instruction and fine-tuning
Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multilingual federated prompt tuning, training only prompt encoders (or LoRA) on XLM-R base with FedAvg-style aggregation across language clients, evaluated on NC, XNLI and MasakhaNEWS for low-resource languages.
Problem
Fine-tuning multilingual LLMs for low-resource languages is hindered by data-sharing restrictions across regions, linguistic differences (non-IID), limited compute, and scarce data.
Approach
Combine federated averaging with parameter-efficient prompt tuning so each language/country client trains only a small virtual-token prompt encoder while the PLM stays frozen, and analyze gains via a language-distance metric defined as negative log cosine similarity to a token-weighted average pre-training language vector.
Key results
- On XNLI, federated prompt tuning averages 39.83% accuracy vs 32.94% for PE monolingual fine-tuning.
- On NC, PE_FL (IID) reaches 83.6% average vs 64.3% for PE monolingual.
- Communication reduced from 108 GB to 478.93 MB (about 99%) with a 2x768 prompt encoder under 5 clients and 10 rounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use federated prompt tuning instead of PE_Monolingual | XLM-RoBERTa base · XNLI | average accuracy: 32.94 -> 39.83 | Helped |
| Use federated prompt tuning in the IID setting instead of PE_Centralized | XLM-RoBERTa base · News Classification (NC) | average accuracy: 77.5 -> 83.6 | Helped |
| Use federated prompt tuning in the Non-IID setting instead of PE_Centralized | XLM-RoBERTa base · News Classification (NC) | average accuracy: 77.5 -> 80.7 | Helped |
| Use federated prompt tuning instead of PE_Centralized | XLM-RoBERTa base · MasakhaNEWS | average accuracy: 79.3 -> 76.4 | Hurt |
| Use federated LoRA instead of PE_Centralized | XLM-RoBERTa base · MasakhaNEWS | average accuracy: 79.3 -> 81.0 | Helped |
| Reduce German NC training data from 8,000 to 30 examples (ablation) | Federated prompt tuning and local monolingual fine-tuning · German News Classification | accuracy | Mixed |
| Apply federated prompt tuning rather than local fine-tuning across languages with greater distance from the pre-trained language (ablation) | XLM-RoBERTa base · News Classification and XNLI | accuracy | Helped |
Surprising findings
- Federated prompt tuning on some languages outperforms the centralized approach, which the authors attribute to implicit regularization.
Limitations
- No added privacy protection (differential privacy or secure aggregation) against attacks such as gradient inversion.
- Evaluated only on a base-sized model (XLM-RoBERTa-base) and classification/NLI tasks; future scale study planned.
- Some results show PE_FL underperforming centralized or LoRA variants on specific languages (e.g., Hausa, Yoruba on MasakhaNEWS).
Experimental setup
Datasets: XGLUE News Classification XNLI MasakhaNEWS UN Corpus
Benchmarks: XGLUE NC XNLI MasakhaNEWS
Metrics: accuracy communication cost (MB/GB) trainable parameter fraction
Baselines: Monolingual fine-tuning Centralized fine-tuning Federated full fine-tuning Federated LoRA PE_Monolingual PE_Centralized
Builds on: FedAvg prompt tuning LoRA XLM-RoBERTa
Closest papers by meaning
- Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models (ICML 2025 · similarity 0.89)
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.89)
- FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models (ICML 2024 · similarity 0.89)
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.89)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.89)
- FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.