Research map / Federated learning and privacy / Personalized and heterogeneous FL

F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics

CVPR 2025 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes F3OCUS, a federated PEFT method that selects VLM layers per client via LNTK principal-eigenvalue importance and refines the selection on the server with multi-objective meta-heuristics, and releases the MedVQA-FL / Ultra-MedVQA dataset.

Problem

Parameter-efficient fine-tuning of large vision-language models on resource-constrained, heterogeneous clients in federated learning, where naive client-local layer selection ignores other clients and slows convergence.

Approach

Client-side layer importance from the principal eigenvalue of layerwise Neural Tangent Kernels, plus a data-free server-side multi-objective meta-heuristic that jointly maximizes cumulative importance and minimizes variance of layer-selection counts across clients.

Key results

• F3OCUS improves over the closest pruning baseline by 5.35% (homogeneous) and 5.33% (heterogeneous) on VQA tasks, per the paper's summary across architectures.
• On Task 1 with ALBEF, F3OCUS reaches 42.78 overall score with 0.30M trainable params and 9.7 MBits communication, versus 43.17 for all-adapter tuning at 28.8 MBits.
• Across 6 FL task settings, 4 VLMs and 58 medical datasets, the paper reports more than 10,000 client-level experiments.

Measured effects reported in the paper

Surprising findings

• Adapting all adapters (FedAvg-style full adapter tuning) outperforms F3OCUS on the ALBEF Task 1 overall score, although F3OCUS uses far less communication.

Limitations

• Server-side meta-heuristic search adds optimization over five algorithms; results depend on algorithm choice and budget per client.
• Convergence analysis yields only convergence to a neighbourhood of a stationary point with a non-zero error floor.

Experimental setup

Datasets: SLAKE VQA-RAD VQA-Med 2019 VQA-Med 2020 VQA-Med 2021 Open-I MIMIC CT Ultrasound Dermatoscopy Fundus Histology

Benchmarks: SLAKE VQA-RAD VQA-Med 2019 VQA-Med 2020 VQA-Med 2021 Ultra-MedVQA Open-I MIMIC

Metrics: accuracy F1-score communication MBits GFLOPs trainable parameters (M)

Baselines: FedDAT LoRA FedAvg adapters (All adapters) Federated Dropout Magnitude pruning FishMask GradFlow GraSP SNIP Synflow RGN Fedselect

Builds on: FedAvg Neural Tangent Kernel Adapter tuning SNIP FedDAT

Closest papers by meaning

• Enhancing Foundation Models with Federated Domain Knowledge Infusion (ICML 2025 · similarity 0.89)
• FedPIA – Permuting and Integrating Adapters Leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning (AAAI 2025 · similarity 0.89)
• FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.89)
• FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.88)
• BLEND: Balancing Personalization vs. Generalization in Federated Vision–Language Models (NeurIPS 2026 · similarity 0.88)
• Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.88)