Research map / Federated learning and privacy / Personalized and heterogeneous FL
Federated Model Heterogeneous Matryoshka Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedMRL, which adds a shared small homogeneous model to each client's heterogeneous model, fuses their representations with a personalized projector, and trains multi-granularity Matryoshka representations across both heads for model-heterogeneous federated learning.
Problem
Model heterogeneous federated learning where clients train heterogeneous local models; existing mutual-learning methods transfer only limited knowledge via output-level losses, causing performance bottlenecks and communication/computation costs.
Approach
Add a global homogeneous small model per client alongside the heterogeneous local model; splice their feature-extractor outputs, fuse them with a personalized linear projector, and build Matryoshka representations at two granularities that are classified by the global header and the local header, so both models are trained jointly on the summed loss.
Key results
- Under model-heterogeneous FL with N=100, C=10%, FedMRL reaches 95.85% (CIFAR-10) and 62.15% (CIFAR-100) average test accuracy, beating the best baseline FedProto (92.49%, 53.67%).
- Improvements over the best baseline reach up to 8.48% and over the best same-category mutual learning baseline up to 24.94%.
- Theoretical analysis gives a non-convex convergence rate of O(1/T).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedMRL instead of the strongest baseline in each model-heterogeneous setting | Model-heterogeneous FL · CIFAR-10 and CIFAR-100, six client-participation settings | average test accuracy (%): up to 8.48% | Helped |
| Use FedMRL instead of the strongest mutual-learning baseline in each model-heterogeneous setting | Model-heterogeneous FL · CIFAR-10 and CIFAR-100, six client-participation settings | average test accuracy (%): up to 24.94% | Helped |
| Use FedMRL instead of the strongest baseline in each model-homogeneous setting | Model-homogeneous FL · CIFAR-10 and CIFAR-100, six client-participation settings | average test accuracy (%): up to 5.71% | Helped |
| Add Matryoshka representations and the global header (ablation) | FedMRL · CIFAR-10 and CIFAR-100, N=100, C=10% | average test accuracy (%) | Helped |
| Reduce the homogeneous small model representation dimension d1 | FedMRL · CIFAR-10 and CIFAR-100, N=100, C=10% | average test accuracy (%) | Helped |
| Reduce the homogeneous small model representation dimension d1 | FedMRL · CIFAR-10 and CIFAR-100 | communication cost | Helped |
| Reduce the homogeneous small model representation dimension d1 | FedMRL · CIFAR-10 and CIFAR-100 | computation overhead | Helped |
| Use FedMRL instead of FedProto | Model-heterogeneous FL · CIFAR-10 and CIFAR-100, N=100, C=10% | computation cost to target accuracy | Helped |
| Use FedMRL instead of FedProto | Model-heterogeneous FL · CIFAR-10 and CIFAR-100 | communication cost | Hurt |
| Use mix-small instead of mix-large for inference | FedMRL · CIFAR-10 and CIFAR-100, N=100, C=10% | accuracy | No clear effect |
Surprising findings
- Mutual-learning baselines such as FML often fail to converge in the model-heterogeneous setting (e.g., 30.48% on CIFAR-10 with N=10).
Limitations
- Global header adds storage, communication and training overhead even though it is one linear layer.
- Evaluated only on two small CIFAR image classification benchmarks with CNN models.
- Per-round computation is higher than baselines because of the extra homogeneous model and projector.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: average test accuracy communication rounds communication cost (parameters) computation FLOPs
Baselines: Standalone LG-FedAvg FD FedProto FML FedKD FedAPEN
Builds on: Matryoshka Representation Learning FedAvg FML FedKD FedAPEN
Closest papers by meaning
- One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models (ICML 2025 · similarity 0.89)
- A Swiss Army Knife for Heterogeneous Federated Learning: Flexible Coupling via Trace Norm (NeurIPS 2024 · similarity 0.88)
- Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.88)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.88)
- FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extraction (NeurIPS 2024 · similarity 0.88)
- FedSA: A Unified Representation Learning via Semantic Anchors for Prototype-based Federated Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.