Research map / Federated learning and privacy / Personalized and heterogeneous FL
Towards Multi-modal Transformers in Federated Learning
Guangyu Sun, Matias Mendieta, Aritra Dutta, Xin Li, Chen Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add collaborative aggregation (in-modal collaboration on self-attention layers) (ablation) | FedCola · Flickr10k default | sum of top-1 recalls: to 88.70 | Improved |
| add compensation scheme to fix layer and modality misalignment (ablation) | FedCola · Flickr10k default | sum of top-1 recalls: 88.70 -> 90.09 | Improved |
| add complementary local training with gated cross-modal weights (ablation) | FedCola · Flickr10k default | sum of top-1 recalls: 90.09 -> 91.96 | Improved |
| make out-modality weights trainable instead of frozen in complementary local training (ablation) | FedCola · Flickr10k default | sum of top-1 recalls | Improved |
| use only self-attention layers instead of whole transformer blocks for complementary local training (ablation) | FedCola · Flickr10k default | retrieval vs communication cost | Mixed |
| collaborate on self-attention layers instead of MLPs or whole blocks (ablation) | FedCola aggregation · Flickr10k default | sum of top-1 recalls | Improved |
| collaborate within the same modality instead of across all models (ablation) | FedCola aggregation · Flickr10k default | sum of top-1 recalls | Improved |
| use FedProx instead of FedAvg with transformer backbone | FedAvg · Flickr10k default | sum of top-1 recalls: 81.08 -> 78.55 | Hurt |
| increase domain gap between uni-modal and multi-modal clients | FedCola · domain gap settings | relative gain over FedAvg | Hurt |
| add more uni-modal datasets (CIFAR-100, AG NEWS, OrganCMNIST, Medical Abstracts) | FedCola · multi-modal evaluation | retrieval recall | Improved |
Closest papers by meaning
- FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.88)
- Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data (NeurIPS 2024 · similarity 0.88)
- FedMBridge: Bridgeable Multimodal Federated Learning (ICML 2024 · similarity 0.87)
- Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models (ICML 2024 · similarity 0.87)
- Federated Learning via Input-Output Collaborative Distillation (AAAI 2024 · similarity 0.87)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
