Research map / Federated learning and privacy / Personalized and heterogeneous FL

Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs

NeurIPS 2026 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Semantic Consensus (FedCoFiT), a federated LLM fine-tuning method where clients exchange generated responses on a shared public prompt set, the server clusters them in embedding space and returns a consensus pseudo-label for further local training, cutting communication by orders of magnitude versus LoRA aggregation.

Problem

Parameter-aggregation federated fine-tuning of LLMs requires transmitting large adapter weights, assumes aligned architectures, and needs white-box access, which becomes impractical for large, heterogeneous, or black-box models.

Approach

Collaboration is mediated through model behavior: clients share generated responses on public prompts, the server forms a per-prompt semantic consensus via embedding clustering (density-based) and centroid-nearest representative selection, and broadcasts pseudo-labels that clients use in standard causal LM fine-tuning alongside private data.

Key results

• On LLaMA-7B Dolly IFEval, FedCoFiT scores 54.36 with 0.006 GB communication versus 3.13 GB (FedIT), 3.75 GB (FlexLoRA), and 20.63 GB (FLoRA).
• Analytically, communication for Llama3.1-405B with LoRA on attention+MLP is about 1006x larger than Semantic Consensus (5.2 MiB per round in the stated setting).
• On TinyLlama, FedCoFiT achieves the best Dolly IFEval score (51.19) and the best Wizard MMLU (22.60) among federated methods.

Surprising findings

• FedCoFiT outperforms the federated baselines and even the centralized LoRA reference on some metrics (e.g., LLaMA2 Alpaca IFEval 60.31 vs centralized 59.24), though gaps are small and mostly within noise.

Limitations

• Communication comparison is largely analytic for 405B scale; empirical runs are on TinyLlama, LLaMA-7B and LLaMA2-13B.
• Gains in task quality over baselines are small and mixed; the method sometimes underperforms (e.g., TinyLlama Alpaca IFEval 50.71 vs FLoRA 51.10).
• Convergence and privacy analyses rely on strong assumptions (bounded gradients, smoothness, stationarity) and the DP result is a sketch.

Experimental setup

Datasets: Dolly-15k Alpaca WizardLM OpenOrca ShareGPT UltraChat

Benchmarks: IFEval MMLU MT-Bench

Metrics: IFEval score MMLU score MT-Bench score communication volume (GB) wall-clock runtime energy consumption

Baselines: Centralized (LoRA) FedIT FLoRA FlexLoRA

Builds on: FedAvg LoRA FedIT FLoRA FlexLoRA

Closest papers by meaning

• Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.92)
• Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models (ICML 2025 · similarity 0.92)
• FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.91)
• FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.90)
• FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models (ICML 2024 · similarity 0.90)
• Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.90)