Research map / Federated learning and privacy / Personalized and heterogeneous FL
Adaptive Latent-Space Constraints in Personalized Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds adaptive, kernel-optimized MMD penalties (MK-MMD and MMD-D) on latent features to penalty-based personalized FL methods Ditto and MR-MTL, improving accuracy especially under feature heterogeneity.
Problem
Penalty-based personalized federated learning typically constrains weight drift with an l2 norm, which may not handle feature-level heterogeneity; existing feature-based constraints (contrastive, cosine, prototype) are non-adaptive or limited to classification.
Approach
Replace or augment Ditto's and MR-MTL's weight-drift penalty with an unpaired, distribution-level MMD penalty on latent features, where the kernel (MK-MMD coefficients or a deep MMD-D kernel) is periodically re-optimized to maximize test power during training.
Key results
- On RxRx1, Ditto with MMD-D replacement reaches 67.478 vs 65.629 for standard Ditto; on Synthetic(0.5) MMD-D augmented with Ditto reaches 89.695 vs 85.533 for Ditto.
- Vanilla Ditto outperforms the MMD variants standalone on CIFAR-10 and Fed-ISIC2019 (e.g., Fed-ISIC2019 71.350 Ditto vs 64.302 MMD-D replacement), but MMD-D augmentation gives 72.226 on Fed-ISIC2019.
- MR-MTL augmented with MMD-D improves RxRx1 from 64.065 to 66.673 and CIFAR-10 alpha=0.1 from 79.516 to 80.307 (MK-MMD_20 gives 81.269).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the Ditto weight constraint with an adaptive MMD feature constraint | Ditto · Synthetic datasets and RxRx1, compared with CIFAR-10 and Fed-ISIC2019 | accuracy | Mixed |
| add MMD-D feature constraint to Ditto | Ditto · Fed-ISIC2019 | balanced accuracy: 71.350 -> 72.226 | Helped |
| add MMD-D feature constraint to Ditto | Ditto · RxRx1 | accuracy: 65.629 -> 67.755 | Helped |
| add MMD-D feature constraint to Ditto | Ditto · Synthetic datasets | accuracy | Hurt |
| add MMD feature constraint to MR-MTL | MR-MTL · All reported tasks except Fed-ISIC2019 | accuracy | Helped |
| add MMD-D or MK-MMD feature constraint to MR-MTL | MR-MTL · Fed-ISIC2019 | balanced accuracy | Hurt |
| increase the number of constrained latent layers beyond depth 1 (ablation) | MMD-constrained CIFAR-10 models · CIFAR-10 α=0.1 | accuracy | Hurt |
| optimize the RBF kernel weights rather than use fixed uniformly weighted kernels (ablation) | MK-MMD on Synthetic datasets · Synthetic datasets | accuracy | Helped |
| replace standard Ditto constraint with cosine-similarity feature alignment | Ditto · CIFAR-10 α=5.0 | accuracy: 77.658 -> 67.298 | Hurt |
| add MK-MMD feature constraint to Ditto | Ditto · CIFAR-10 α=0.5 | accuracy: 80.702 -> 80.976 | Helped |
Surprising findings
- Replacing Ditto's weight penalty with feature-based MMD constraints underperforms standard Ditto on CIFAR-10 and Fed-ISIC2019, and adding the Ditto penalty to the Synthetic datasets hurts performance across the board.
- Cosine-similarity feature constraint alone performs worse than Ditto in many settings, e.g., CIFAR-10 alpha=5.0 drops to 67.298 from 77.658 under replacement.
Limitations
- Kernel optimization adds substantial computational cost (e.g., MK-MMD QP solving O(m^3.5); training time up to several-fold longer).
- Results are on small models and four datasets with 3 seeds; benefits are mixed and do not hold uniformly (e.g., Fed-ISIC2019 with MR-MTL and standalone MMD).
- Theoretical conditions under which MMD constraints help are not quantified.
Experimental setup
Datasets: CIFAR-10 Synthetic (extended from prior generator) Fed-ISIC2019 (FLamby) RxRx1
Benchmarks: CIFAR-10 (Dirichlet alpha 0.1, 0.5, 5.0) Synthetic (alpha=beta 0.0, 0.5) Fed-ISIC2019 RxRx1
Metrics: accuracy balanced accuracy (Fed-ISIC2019)
Baselines: FedAvg Ditto Cosine similarity feature penalty MR-MTL MOON FedProto
Builds on: Ditto MR-MTL Multi-Kernel MMD MMD-D FedProx
Closest papers by meaning
- FedAS: Bridging Inconsistency in Personalized Federated Learning (CVPR 2024 · similarity 0.89)
- Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guarantees (ICML 2025 · similarity 0.89)
- Flexible Sharpness-Aware Personalized Federated Learning (AAAI 2025 · similarity 0.89)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.89)
- A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.89)
- Personalized Federated Learning via Feature Distribution Adaptation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.