Research map / Federated learning and privacy / Personalized and heterogeneous FL
Efficient Collaborative LLM Fine-Tuning over Heterogeneous Mobile Devices via Many Backbones to One Side-Network Tuning
Xingke Yang, Liang Li, Sicong Li, Liwei Guan, Hao Wang, Xiaoqi Qin, Jiang Liu, Xin Fu, Miao Pan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a system for collaborative LLM fine-tuning across heterogeneous mobile devices, addressing on-device resource overhead, straggler-prone synchronous aggregation, and the single-shared-backbone assumption.
Problem
Federated LLM fine-tuning over heterogeneous mobile devices faces prohibitive on-device resource overhead, straggler-prone synchronous aggregation, and the assumption of a unified pretrained backbone.
Approach
Uses many backbones mapped to one side-network tuning scheme for collaborative fine-tuning on heterogeneous mobile devices; details beyond the abstract are not available in the provided text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token selection (upload only loss-relevant activations) (ablation) | EC-MobiLLM · MRPC heterogeneous | communication / TTA: 1.1 MB -> 281.6 MB without; TTA 5.9 -> 45.7 min without | Improved |
| add server caching of activations (ablation) | EC-MobiLLM · MRPC heterogeneous | computation / TTA: 2.7 -> 54.0 TFLOPs without; TTA 5.9 -> 91.3 min without | Improved |
| add non-blocking asynchronous collaboration (ablation) | EC-MobiLLM · MRPC heterogeneous | TTA (min): 9.6 -> 5.9 | Improved |
| ablate TS, SC or NBAC (ablation) | EC-MobiLLM · MRPC heterogeneous | accuracy: 88.1 in all variants | No clear effect |
| train shared side-network collaboratively (Global) instead of per-device (Single) (ablation) | EC-MobiLLM · SST-2 | accuracy: TX2 88.4 -> 92.1; AGX 90.0 -> 92.9 | Improved |
| allow device-specific larger backbones instead of shared small backbone with LoRA rank tuning | EC-MobiLLM · SST-2 AGX | accuracy: FLoRA 90.8 -> 92.9 | Improved |
| use average-interval layer sampling instead of importance-based selection (ablation) | EC-MobiLLM · SST-2 alignment study | accuracy | No clear effect |
| use more backbone activation layers (ablation) | EC-MobiLLM · OPT-1.3B/RoBERTa-Large | accuracy | Improved |
| force side-network hidden size to mismatch backbone (e.g. 2048 for OPT-125M) (ablation) | EC-MobiLLM · SST-2 | accuracy | Hurt |
| upload privatized activations once and cache vs repeated releases under same DP budget (ablation) | EC-MobiLLM · SST-2 DP | accuracy: AGX 86.5 -> 89.2 | Improved |
| add DP noise to activations (one-shot) | EC-MobiLLM · SST-2 AGX | accuracy: 92.9 -> 89.2 | Hurt |
| replace LoRA/BitFit federated training with server-side side-network tuning | EC-MobiLLM · MRPC centralized | accuracy: 91.5 (LoRA) -> 89.6 | Hurt |
Closest papers by meaning
- Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024 · similarity 0.89)
- Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.87)
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.87)
- Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models (ICML 2025 · similarity 0.86)
- Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
