Research map / Federated learning and privacy / Personalized and heterogeneous FL
Generating Synthetic Data for Unsupervised Federated Learning of Cross-Modal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedWGAN, which trains per-client WGAN generators with three added losses (dual contrastive, prototype-based JSD alignment) and server-side prototype filtering to create synthetic data that mitigates non-IID issues in unsupervised federated cross-modal hashing.
Problem
Unsupervised federated cross-modal retrieval suffers from non-IID data distribution across clients (varying amounts and categories of data), with no labels available and no sharing of raw client data.
Approach
Train a WGAN generator on each client constrained by cluster-prototype-based pseudo-label contrastive losses and a JSD distribution-alignment loss; generate synthetic data on the server from uploaded generators and filter outliers against global prototypes before training the global cross-modal hashing model.
Key results
- On MIRFLICKR I2T at 16 bits, FedWGAN reaches 0.750 mAP@50 vs 0.741 for the next best (PT-FUCH) and 0.707 for FedAvg baseline.
- On MS COCO I2T 16 bits, FedWGAN reaches 0.684 vs 0.661 for FedCAFE, the largest gain reported.
- Applying the generation method to FedAvg, MOON, PLFedCMH and PEPFCH improves I2T by 1.7% to 7.6% on homogeneous frameworks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedWGAN instead of PT-FUCH | Unsupervised federated cross-modal retrieval · MIRFLICKR, 16-bit I2T | mAP@50: 0.741 -> 0.750 | Helped |
| Use FedWGAN instead of PT-FUCH | Unsupervised federated cross-modal retrieval · MIRFLICKR, 16-bit T2I | mAP@50: 0.726 -> 0.738 | Helped |
| Use FedWGAN instead of PT-FUCH | Unsupervised federated cross-modal retrieval · MS COCO, 16-bit I2T | mAP@50: 0.655 -> 0.684 | Helped |
| Use FedWGAN instead of PT-FUCH | Unsupervised federated cross-modal retrieval · MS COCO, 16-bit T2I | mAP@50: 0.659 -> 0.676 | Helped |
| Use FedWGAN instead of PT-FUCH | Unsupervised federated cross-modal retrieval · NUS-WIDE, 16-bit I2T | mAP@50: 0.633 -> 0.658 | Helped |
| Use FedWGAN instead of PT-FUCH | Unsupervised federated cross-modal retrieval · NUS-WIDE, 16-bit T2I | mAP@50: 0.626 -> 0.652 | Helped |
| Add client WGAN training and server-side synthetic data generation (ablation) | FedAvg · MIRFLICKR, MS COCO, and NUS-WIDE; 16-bit I2T and T2I | mAP@50: 0.707 -> 0.723; 0.634 -> 0.665; 0.590 -> 0.623 (I2T); 0.697 -> 0.704; 0.640 -> 0.662; 0.578 -> 0.616 (T2I) | Helped |
| Add dual contrastive constraints to WGAN (ablation) | FedWGAN w/GAN · MIRFLICKR, MS COCO, and NUS-WIDE; 16-bit I2T and T2I | mAP@50: 0.723 -> 0.741; 0.665 -> 0.672; 0.623 -> 0.635 (I2T); 0.704 -> 0.721; 0.662 -> 0.668; 0.616 -> 0.634 (T2I) | Helped |
| Add JSD data-distribution alignment loss (ablation) | FedWGAN w/cont · MIRFLICKR, MS COCO, and NUS-WIDE; 16-bit I2T and T2I | mAP@50: 0.741 -> 0.746; 0.672 -> 0.680; 0.635 -> 0.648 (I2T); 0.721 -> 0.739; 0.668 -> 0.673; 0.634 -> 0.646 (T2I) | Helped |
| Add global prototype-based synthetic-data filtering (ablation) | FedWGAN w/JSD · MIRFLICKR, MS COCO, and NUS-WIDE; 16-bit I2T and T2I | mAP@50: 0.746 -> 0.750; 0.680 -> 0.684; 0.648 -> 0.658 (I2T); 0.739 -> 0.738; 0.673 -> 0.676; 0.646 -> 0.652 (T2I) | Mixed |
Surprising findings
- Gains from the data generation method are reduced for heterogeneous FL frameworks, which already transfer more data to the server.
Limitations
- Gains shrink as non-IID severity decreases (beta = 1 or 10), with FedCAFE sometimes outperforming FedWGAN at beta=10.
- Heterogeneous FL frameworks see only slight improvements from the method.
Experimental setup
Datasets: MIRFLICKR-25K MS COCO NUS-WIDE
Benchmarks: MIRFLICKR-25K MS COCO NUS-WIDE
Metrics: mAP@50
Baselines: Scaffold FedAvg FedProx MOON FedX FedCAFE PT-FUCH PLFedCMH PEPFCH
Builds on: WGAN UCCH PT-FUCH FedAvg
Closest papers by meaning
- Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains (NeurIPS 2024 · similarity 0.85)
- The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.85)
- Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data (CVPR 2024 · similarity 0.84)
- NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrieval (CVPR 2025 · similarity 0.84)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.84)
- Learning Shared Representations from Unpaired Data (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.