Research map / Federated learning and privacy / Communication-efficient FL
Federated In-Context Learning: Iterative Refinement for Improved Answer Quality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fed-ICL, a parameter-free federated in-context learning framework where clients iteratively refine server queries' answers using local example data, transmitting only query–answer pairs, with a linear self-attention convergence analysis.
Problem
Improving in-context learning answer quality for QA when high-quality examples are scarce and distributed across privacy-constrained clients, without transmitting model parameters.
Approach
Clients use the server's current query answers as in-context examples to relabel their local data, then use both local data and relabeled data as ICL context to answer server queries; the server averages/fuses responses over multiple rounds, so only (query, answer) pairs are communicated and convergence is shown for a linear self-attention model.
Key results
- Fed-ICL consistently outperforms Fed-ICL-GT (single-round variant), showing gains from iterative communication rounds across alpha settings on MMLU and TruthfulQA.
- Fed-ICL-Free still outperforms LLM-Debate, which does not use local datasets, despite lacking ground-truth answers on clients.
- Gains over baselines are clearer for weaker open-source backbones (Llama) than for GPT-4o-mini, per the paper's analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iterative client-server refinement instead of one-shot interaction (ablation) | Fed-ICL · MMLU and TruthfulQA | answer quality | Helped |
| Increase the number of interaction rounds (ablation) | Fed-ICL and Fed-ICL-Free · MMLU | performance | Helped |
| Increase clients from three to five (ablation) | Fed-ICL · MMLU | performance | Hurt |
| Increase context length from 1 to 5 examples (ablation) | Fed-ICL and Fed-ICL-Free · MMLU | response accuracy | Helped |
| Apply local nearest-neighbor filtering to select semantically relevant examples (ablation) | Fed-ICL and Fed-ICL-Free · MMLU | performance | Helped |
| Use inferred local examples without ground-truth answers (ablation) | Fed-ICL-Free · MMLU and TruthfulQA | answer quality | Helped |
| Use local client datasets and iterative refinement rather than LLM-Debate | Fed-ICL · TruthfulQA | answer quality | Helped |
| Use Fed-ICL with open-source Llama backbones rather than other baselines | Fed-ICL · MMLU and TruthfulQA | performance | Helped |
| Use Fed-ICL with GPT-family backbones rather than other baselines | Fed-ICL · MMLU and TruthfulQA | performance | Mixed |
Surprising findings
- Performance gains over baselines are less pronounced with stronger GPT-family backbones, suggesting FL collaboration helps weaker LMs most.
Limitations
- Theory covers only a single-layer linear self-attention model on regression tasks, not nonlinear transformers or QA generation.
- Performance degrades as the number of clients grows due to data heterogeneity.
- Gains are smaller for stronger GPT-family backbones.
Experimental setup
Datasets: MMLU TruthfulQA
Benchmarks: MMLU TruthfulQA
Metrics: accuracy BERTScore BLEURTScore BARTScore communication cost
Baselines: FedAvg LLM-Debate MoA LLM-Blender Fed-ICL-GT Fed-ICL-UB Fed-ICL-LB
Builds on: FedAvg LLM-Debate Mixture-of-Agents LLM-Blender OpenFedLLM
Closest papers by meaning
- Code-Style In-Context Learning for Knowledge-Based Question Answering (AAAI 2024 · similarity 0.87)
- Implicit In-context Learning (ICLR 2025 · similarity 0.87)
- FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024 · similarity 0.86)
- How to Configure Good In-Context Sequence for Visual Question Answering (CVPR 2024 · similarity 0.86)
- LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.85)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.