Research map / Federated learning and privacy / Personalized and heterogeneous FL

BLEND: Balancing Personalization vs. Generalization in Federated Vision–Language Models

NeurIPS 2026 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BLEND, a federated CLIP fine-tuning framework that aggregates global vision/text adapters, keeps a personalized vision projection adapter local, and trains with a fusion loss plus a running anchor loss to balance personalization and generalization.

Problem

Federated fine-tuning of vision-language models under non-IID client data must balance personalization to local classes against generalization to unseen classes; existing FedVLM methods trade one off against the other.

Approach

Selectively aggregate global vision and text adapters while keeping a personalized projection adapter local; fuse a frozen-head generalized vision embedding with a personalized embedding via a weighted average, and regularize with an anchor loss tying the generalized vision embedding to the zero-shot CLIP text embeddings.

Key results

• BLEND achieves cross-dataset average H-Mean of 82.15% vs 77.34% for the best baseline MMA+FedAvg (+4.81) with 10 clients over 50 rounds.
• On DTD, BLEND reaches 75.59 H-Mean vs 68.56 for MMA+FedAvg; on FGVC Aircraft it reaches 45.09 vs 36.10.
• The average H-Mean gain over the strongest baseline grows to +5.64 at N=20 and +6.11 at N=30 clients (as reported in the appendix).

Surprising findings

• Aggregating all trainable parameters ('Full') improves generalization slightly but substantially harms personalization.
• Unfreezing the text projection head mostly degrades H-Mean due to forgetting pre-trained CLIP knowledge.

Limitations

• Built only on CLIP and text-image modalities; no extension to audio/video, generation, or retrieval.
• Performance depends on hyperparameters gamma and lambda whose adaptive selection is open.
• No privacy mechanism (e.g., differential privacy) and not decentralized/serverless.

Experimental setup

Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 DTD EuroSAT FGVC Aircraft UCF101 iNaturalist

Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 DTD EuroSAT FGVC Aircraft UCF101

Metrics: personalization accuracy generalization accuracy harmonic mean cross-dataset std

Baselines: Zero-shot CLIP FedOTP PromptFolio FedPGP PromptFL pFedMMA pFedMoAP FedPHA MMA+FedAvg

Builds on: CLIP MMA FedAvg PromptFL

Closest papers by meaning

• Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.90)
• Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.88)
• F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics (CVPR 2025 · similarity 0.88)
• Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method (NeurIPS 2024 · similarity 0.87)
• FedPIA – Permuting and Integrating Adapters Leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning (AAAI 2025 · similarity 0.86)
• Enhancing Foundation Models with Federated Domain Knowledge Infusion (ICML 2025 · similarity 0.86)