Research map / Federated learning and privacy / Personalized and heterogeneous FL
Federated Representation Learning in the Under-Parameterized Regime
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLUTE, a federated representation learning algorithm for the under-parameterized regime with a regularized loss and server-side joint updates of representation and heads, with linear-model sample complexity and convergence guarantees plus CIFAR experiments.
Problem
Federated representation learning when the model class is too small to express all clients' ground-truth models (under-parameterized regime), where naive averaging of local representations fails to recover the optimal shared subspace.
Approach
Adds two regularizers (penalizing ||BW||_F^2 and a balanced term ||B^T B||_F^2 + ||W W^T||_F^2) to the ERM objective, uses random initialization, and has the server jointly update the representation and all local heads with data-free penalty gradients to distill the top-k subspace of the ground-truth matrix.
Key results
- On CIFAR-10 (100x2 partition), FLUTE achieves 86.128% vs FedRep 86.685%, while on 100x5 it reaches 76.918% vs FedRep 73.808%.
- On CIFAR-100 100x5, FLUTE reaches 77.750% vs FedRep 78.621%; the variant FLUTE* reaches 79.560%, outperforming baselines.
- Linear sample complexity is O~(max{d,M}/(M eps^2)), giving linear speedup in M when d>=M, and exponential convergence to the optimal model with sufficient samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FLUTE instead of the strongest listed baseline | CNN classification · CIFAR-10 partitions | test accuracy: 87.320{ $\pm$1.485} -> 87.012{ $\pm$0.453}; 75.766{ $\pm$0.220} -> 76.478{ $\pm$0.484}; 87.177{ $\pm$0.489} -> 86.128{ $ | Mixed |
| Use FLUTE instead of the strongest listed baseline | CNN classification · CIFAR-100 partitions | test accuracy: 78.892{ $\pm$0.410} -> 77.750{ $\pm$0.615}; 68.630{ $\pm$0.705} -> 70.598{ $\pm$0.282}; 57.280{ $\pm$0.105} -> 59.243{ $ | Mixed |
| Increase local updates and perform 10 head-only updates after the joint update (ablation) | FLUTE · CIFAR-10 partitions | test accuracy: 87.012{ $\pm$0.453} -> {87.713{ $\pm$1.365}}; 76.478{ $\pm$0.484} -> {76.543{ $\pm$0.921}}; 86.128{ $\pm$1.007} -> {88.5 | Helped |
| Increase local updates and perform 10 head-only updates after the joint update (ablation) | FLUTE · CIFAR-100 partitions | test accuracy: 77.750{ $\pm$0.615} -> {79.560{ $\pm$0.627}}; 70.598{ $\pm$0.282} -> {70.844{ $\pm$0.419}}; 59.243{ $\pm$0.334} -> {59.7 | Helped |
Surprising findings
- Randomly initialized FLUTE is slower early on than FedRep, which uses a data-based covariance initialization, but eventually outperforms it.
- FLUTE's advantage over baselines is not uniform; on some CIFAR-10 partitions (e.g., 100x2) FedRep and FedRep* are comparable or better.
Limitations
- Theory is limited to linear models with Gaussian-like data and requires sufficiently large per-client sample sizes.
- Clients transmit local heads as well as representations, increasing communication cost compared with FedRep-style methods.
- Sample complexity depends on the eigengap Delta, and the required samples grow exponentially in the number of iterations.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 synthetic Gaussian linear data
Benchmarks: CIFAR-10 CIFAR-100 synthetic linear regression
Metrics: average test accuracy average representation/head error
Baselines: FedAvg FedAvg-FT Fed-LG FedPer FedRep FedRod FedCP
Builds on: FedRep FedPer low-rank matrix factorization gradient descent analysis FedRod FedCP
Closest papers by meaning
- PeFLL: Personalized Federated Learning by Learning to Learn (ICLR 2024 · similarity 0.87)
- Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data (CVPR 2024 · similarity 0.87)
- Federated Model Heterogeneous Matryoshka Representation Learning (NeurIPS 2024 · similarity 0.86)
- Convergence Analysis of Split Federated Learning on Heterogeneous Data (NeurIPS 2024 · similarity 0.86)
- FedLWS: Federated Learning with Adaptive Layer-wise Weight Shrinking (ICLR 2025 · similarity 0.86)
- FedAS: Bridging Inconsistency in Personalized Federated Learning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.