Research map / Federated learning and privacy / Personalized and heterogeneous FL
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pilot, a federated multimodal instruction tuning framework that uses two-stage "adapter on adapter" connectors (task/client-specific adapters plus a cross-task MoA) and Euclidean-distance adaptive aggregation of LoRA text adapters on LLaVA-1.5.
Problem
Federated instruction tuning of multimodal LLMs across clients holding heterogeneous multimodal tasks (captioning, VQA, grounding), where naive FedIT aggregation causes parameter conflicts and degrades local performance.
Approach
Stacks task-specific and client-specific adapters in the vision-LLM connector, then adds a cross-task Mixture-of-Adapters with cross-task adapters and a router so clients absorb other tasks' knowledge; text (LoRA) adapters are aggregated with weights from Euclidean distance over the top-M nearest clients, while visual adapters are aggregated per task.
Key results
- In the FL visual understanding scenario, the full Pilot model reaches 50.4 GQA, 124.0 COCO CIDEr and 51.0 RefCOCO IoU, outperforming local training and FedDPA.
- In the general VQA scenario, Pilot scores 49.2/52.8/52.3 on GQA clients and 54.7/53.2/53.7 on ScienceQA clients, above FedDPA and FedAdam on all nine clients.
- FedAvg underperforms local training on all tasks, e.g. COCO clients drop from about 120 (local) to 86-104 under FedAvg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Pilot rather than the compared federated baselines | LLaVA 1.5 in federated multimodal instruction tuning · FL-oriented visual understanding and general VQA scenarios | task scores | Helped |
| Replace Pilot with FedAvg's aggregation of all training parameters | local models in FedMIT · federated multimodal instruction tuning | task performance | Hurt |
| Add adaptive text-adapter aggregation (ATA) (ablation) | Pilot · GQA, COCO, and RefCOCO | task score: 46.9 -> 47.5; 105.7 -> 110.6; 30.6 -> 39.8 | Helped |
| Add cross-task adapters (ablation) | Pilot with ATA · GQA, COCO, and RefCOCO | task score: 47.5 -> 49.3; 110.6 -> 114.5; 39.8 -> 47.5 | Helped |
| Add router load-balancing and z-loss auxiliary losses (ablation) | Pilot with ATA and cross-task adapters · GQA, COCO, and RefCOCO | task score: 49.3 -> 49.8; 114.5 -> 120.7; 47.5 -> 48.2 | Helped |
| Add the difference loss to separate client-specific and task-specific features (ablation) | Pilot with ATA, cross-task adapters, and router auxiliary losses · GQA, COCO, and RefCOCO | task score: 49.8 -> 50.4; 120.7 -> 124.0; 48.2 -> 51.0 | Helped |
| Train CLIP's MLP layers as CT-CLIP | Pilot · GQA, COCO, and RefCOCO | task score: 50.4 -> 51.6; 124.0 -> 126.1; 51.0 -> 52.7 | Helped |
| Train CLIP's MLP layers as CT-CLIP | Pilot · federated visual understanding efficiency | communication parameters: +0.2B | Hurt |
| Train CLIP's MLP layers as CT-CLIP | Pilot · federated visual understanding efficiency | local client activation parameters: +1.25B | Hurt |
| Aggregate all clients' text-adapter parameters instead of using distance-based Top-M aggregation (ablation) | Pilot · GQA, COCO, and RefCOCO | task score: 50.4 -> 47.9; 124.0 -> 109.7; 51.0 -> 45.2 | Hurt |
| Aggregate only text-adapter parameters from same-task clients instead of using Pilot aggregation (ablation) | Pilot · GQA, COCO, and RefCOCO | task score: 50.4 -> 49.6; 124.0 -> 120.2; 51.0 -> 50.3 | Hurt |
| Set Top-M to 5 instead of 6 (ablation) | Pilot text-adapter aggregation · GQA, COCO, and RefCOCO | task score: 50.4 -> 49.7; 124.0 -> 122.8; 51.0 -> 50.6 | Mixed |
Surprising findings
- Naive FedAvg performs worse than local training on all tasks, indicating multimodal task heterogeneity causes harmful parameter conflicts.
- Shepherd, which aggregates only LLM parameters, underperforms FedAvg, suggesting visual-side parameters matter for FedMIT.
Limitations
- Experiments use only 9 clients and 3 tasks with two scenarios, a small-scale setup.
- Cross-task adapter cost grows with the number of tasks, as the authors note for the CT-CLIP variant.
- Hyperparameters (top-M, lambda values, rounds) are tuned on the reported setting with limited sensitivity analysis.
Experimental setup
Datasets: GQA COCO Caption RefCOCO ScienceQA OCRVQA
Benchmarks: GQA COCO Caption RefCOCO ScienceQA OCRVQA
Metrics: accuracy CIDEr IoU@0.5
Baselines: FedAvg FedProx FedAdam Shepherd FedDPA Local training Centralized training
Builds on: FedAvg FedDPA Shepherd LLaVA-1.5 LoRA
Closest papers by meaning
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.88)
- DISTMOE: Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning (NeurIPS 2026 · similarity 0.87)
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.86)
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.86)
- Re-Imagining Multimodal Instruction Tuning: A Representation View (ICLR 2025 · similarity 0.86)
- Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.