Research map / Federated learning and privacy / Communication-efficient FL
Can Textual Gradient Work in Federated Learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedTextGrad, a federated variant of TextGrad where clients upload locally optimized textual prompts and the server aggregates them via concatenation or LLM summarization, with a Uniform Information Density-guided summarization to improve aggregation.
Problem
Whether textual gradient-based prompt optimization (TextGrad) can be carried out in federated learning settings with black-box LLM APIs, where numerical gradients and losses are unavailable, and how to aggregate distributed textual prompt updates without exceeding context limits or losing informatio
Approach
Replace parameter/gradient aggregation in FL with aggregation of natural-language prompts: clients run TextGrad locally, the server concatenates or LLM-summarizes prompts, and the summarization is guided by the Uniform Information Density principle (measured via GPT-2 surprisal variance) to reduce information non-uniformity.
Key results
- Centralized TextGrad with GPT-4 reaches 0.99 accuracy on BBH Object Counting, and FedTextGrad with GPT-4 reaches 0.98 in the federated setting.
- Concatenation outperforms summarization on BBH Multistep Arithmetic (0.69 vs 0.55), while on Object Counting the gap is small (0.90 vs 0.88).
- UID summarization improves over plain summarization across batch sizes in the heterogeneous setting (e.g., 0.79 vs 0.78 at B=3), with gains that are small and within reported standard deviations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FedTextGrad with summarization instead of centralized TextGrad | GPT-4o · Spatial | accuracy: 0.53 -> 0.40 | Hurt |
| use FedTextGrad with summarization instead of centralized TextGrad | GPT-4o · Web of Lies | accuracy: 0.37 -> 0.30 | Hurt |
| use FedTextGrad with summarization instead of centralized TextGrad | GPT-4o · Zebra Puzzle | accuracy: 0.33 -> 0.27 | Hurt |
| use FedTextGrad with summarization instead of centralized TextGrad | GPT-4o · AMPS Hard | accuracy: 0.46 -> 0.50 | Helped |
| replace concatenation with summarization (ablation) | FedTextGrad · BBH Object Counting | accuracy: 0.90 -> 0.88 | Hurt |
| replace concatenation with summarization (ablation) | FedTextGrad · BBH Multistep Arithmetic | accuracy: 0.69 -> 0.55 | Hurt |
| replace concatenation with summarization (ablation) | FedTextGrad · GSM8K | accuracy: 0.94 -> 0.92 | Hurt |
| transfer a prompt optimized on LLaMA 3.2-11B to LLaMA 3.2-3B | LLaMA 3.2-3B · Object Counting | accuracy: 0.66 -> 0.69 (+0.03) | Helped |
| transfer a prompt optimized on LLaMA 3.2-11B to LLaMA 3.2-3B | LLaMA 3.2-3B · Multi-step Arithmetic | accuracy: 0.51 -> 0.66 (+0.15) | Helped |
| transfer a prompt optimized on LLaMA 3.2-11B to LLaMA 3.2-3B | LLaMA 3.2-3B · GSM8K | accuracy: 0.80 -> 0.72 (-0.08) | Hurt |
| add Uniform Information Density principles to summarization (ablation) | FedTextGrad · Heterogeneous tasks, B = 1 | accuracy: 0.73 (0.03) -> 0.75 (0.02) | Helped |
| add Uniform Information Density principles to summarization (ablation) | FedTextGrad · Heterogeneous tasks, B = 3 | accuracy: 0.78 (0.02) -> 0.79 (0.02) | Helped |
Surprising findings
- Federated prompt optimization on AMPS Hard slightly outperforms centralized TextGrad (0.50 vs 0.46), while it underperforms on the LiveBench reasoning tasks.
- Concatenated prompts grow rapidly with client count and can exceed GPT-4's 8192-token context window, making naive concatenation impractical.
Limitations
- Experiments use small, fixed numbers of clients (typically 3) and a few tasks, with results sometimes within reported standard deviations.
- Summarization degrades performance on complex tasks, and FedTextGrad underperforms centralized TextGrad on several reasoning tasks.
- Privacy, robustness, and communication efficiency are discussed only as open challenges rather than evaluated.
Experimental setup
Datasets: BBH Object Counting BBH Multistep Arithmetic GSM8K LiveBench (Web of Lies, Zebra Puzzle, Spatial) AMPS Hard
Benchmarks: BBH Object Counting BBH Multistep Arithmetic GSM8K LiveBench Web of Lies LiveBench Zebra Puzzle LiveBench Spatial AMPS Hard
Metrics: accuracy
Baselines: TextGrad (centralized) concatenation aggregation summarization aggregation
Builds on: TextGrad FedAvg Uniform Information Density hypothesis
Closest papers by meaning
- Thinking Forward: Memory-Efficient Federated Finetuning of Language Models (NeurIPS 2024 · similarity 0.88)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.87)
- Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.86)
- AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems (NeurIPS 2026 · similarity 0.86)
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.86)
- metaTextGrad: Automatically optimizing language model optimizers (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.