AI Paper Atlas

Research map / Federated learning and privacy / Personalized and heterogeneous FL

Towards Multi-modal Transformers in Federated Learning

Guangyu Sun, Matias Mendieta, Aritra Dutta, Xin Li, Chen Chen

ECCV 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models · 27 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add collaborative aggregation (in-modal collaboration on self-attention layers) (ablation)FedCola · Flickr10k defaultsum of top-1 recalls: to 88.70Improved
add compensation scheme to fix layer and modality misalignment (ablation)FedCola · Flickr10k defaultsum of top-1 recalls: 88.70 -> 90.09Improved
add complementary local training with gated cross-modal weights (ablation)FedCola · Flickr10k defaultsum of top-1 recalls: 90.09 -> 91.96Improved
make out-modality weights trainable instead of frozen in complementary local training (ablation)FedCola · Flickr10k defaultsum of top-1 recallsImproved
use only self-attention layers instead of whole transformer blocks for complementary local training (ablation)FedCola · Flickr10k defaultretrieval vs communication costMixed
collaborate on self-attention layers instead of MLPs or whole blocks (ablation)FedCola aggregation · Flickr10k defaultsum of top-1 recallsImproved
collaborate within the same modality instead of across all models (ablation)FedCola aggregation · Flickr10k defaultsum of top-1 recallsImproved
use FedProx instead of FedAvg with transformer backboneFedAvg · Flickr10k defaultsum of top-1 recalls: 81.08 -> 78.55Hurt
increase domain gap between uni-modal and multi-modal clientsFedCola · domain gap settingsrelative gain over FedAvgHurt
add more uni-modal datasets (CIFAR-100, AG NEWS, OrganCMNIST, Medical Abstracts)FedCola · multi-modal evaluationretrieval recallImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy