Research map / Federated learning and privacy / Personalized and heterogeneous FL
Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoPreFL, a MAML-based distributed pre-training method that meta-optimizes a global model with a loss combining average query loss and cross-client variance, so that federated downstream tasks start from a robust initialization.
Problem
Centrally pre-trained initializations for federated learning generalize poorly to arbitrary downstream FL tasks with unseen labels and heterogeneous clients, yielding limited average accuracy and high cross-client accuracy variance.
Approach
Simulate downstream FL inside pre-training: run FedAvg-style local updates on support sets, then meta-update the aggregated global model on held-out query sets using a meta-loss that mixes average query loss with its cross-client variance (balancer gamma); a hybrid variant also uses server-held data partitioned across pseudo-clients.
Key results
- In scenario I on CIFAR-100 with non-IID downstream FedAvg, CoPreFL reaches 83.29% accuracy and variance 34.69 versus 78.96 / 64.80 for FedAvg-style initialization and 82.45 / 48.72 for FedMeta.
- In scenario II on CIFAR-100, CoPreFL reaches 86.63% accuracy with variance 31.58, outperforming CoPreFL-SGD (83.63 / 41.73) and other baselines.
- On PACS domain shift (Sketch held out), CoPreFL attains 66.83% accuracy and variance 24.31, versus 62.23 / 51.29 for FedAvg pre-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoPreFL pre-training rather than FedMeta pre-training | ResNet-18 initialization for downstream FedAvg · CIFAR-100, scenario I, non-IID | accuracy: 82.45 $\pm$ 3.07 -> 83.29 $\pm$ 2.61 | Helped |
| Use CoPreFL pre-training rather than FedMeta pre-training | ResNet-18 initialization for downstream FedAvg · CIFAR-100, scenario I, non-IID | variance: 48.72 $\pm$ 2.84 -> 34.69 $\pm$ 3.17 | Helped |
| Use CoPreFL pre-training rather than q-FFL pre-training | ResNet-18 initialization for downstream FedAvg · Tiny-ImageNet, scenario I, non-IID | accuracy: 84.11 $\pm$ 2.49 -> 85.23 $\pm$ 2.43 | Helped |
| Use CoPreFL pre-training rather than FedMeta pre-training | ResNet-18 initialization for downstream FedAvg · Tiny-ImageNet, scenario I, non-IID | variance: 37.58 $\pm$ 3.00 -> 35.40 $\pm$ 2.75 | Helped |
| Use CoPreFL rather than FedAvg pre-training | ResNet-18 initialization for downstream FedAvg · CIFAR-100, scenario II, non-IID | accuracy: 82.82 $\pm$ 3.17 -> 86.63 $\pm$ 2.93 | Helped |
| Use CoPreFL rather than FedMeta pre-training | ResNet-18 initialization for downstream FedAvg · CIFAR-100, scenario II, non-IID | variance: 48.44 $\pm$ 2.99 -> 31.58 $\pm$ 2.64 | Helped |
| Use CoPreFL rather than CoPreFL-SGD; apply meta-updates to server data (ablation) | CoPreFL pre-training · CIFAR-100, scenario II, non-IID | accuracy: 83.63 $\pm$ 3.00 -> 86.63 $\pm$ 2.93 | Helped |
| Use CoPreFL rather than CoPreFL-SGD; apply meta-updates to server data (ablation) | CoPreFL pre-training · CIFAR-100, scenario II, non-IID | variance: 41.73 $\pm$ 2.85 -> 31.58 $\pm$ 2.64 | Helped |
| Increase γ from 0.0 to 1.0 (ablation) | CoPreFL pre-training · Tiny-ImageNet, scenario I, non-IID | accuracy: 83.11 $\pm$ 2.17 -> 86.33 $\pm$ 1.92 | Helped |
| Increase γ from 0.0 to 1.0 (ablation) | CoPreFL pre-training · Tiny-ImageNet, scenario I, non-IID | variance: 24.70 $\pm$ 1.95 -> 39.81 $\pm$ 2.30 | Hurt |
| Use CoPreFL rather than centralized pre-training | ResNet-18 initialization for CIFAR-100 downstream FedAvg · CIFAR-100 downstream, ImageNet_1K pre-training | accuracy: 86.75 $\pm$ 2.89 -> 87.96 $\pm$ 1.95 | Helped |
| Use CoPreFL rather than centralized pre-training | ResNet-18 initialization for CIFAR-100 downstream FedAvg · CIFAR-100 downstream, ImageNet_1K pre-training | variance: 67.34 $\pm$ 2.17 -> 30.79 $\pm$ 2.79 | Helped |
Surprising findings
- Centralized pre-training improves average accuracy over random initialization but yields much higher cross-client variance (69.44 vs 41.39 for random init).
- Applying SGD on server data after meta-updating (CoPreFL-SGD) hurts results relative to meta-updating on server data, which the authors attribute to diverting from the balanced objective.
Limitations
- No theoretical analysis linking pre-training strategy to downstream performance (acknowledged as open).
- Experiments limited to image classification with ResNet-18 and 5-way or 20-way downstream tasks on small datasets.
- Gains depend on a balancer gamma that trades average accuracy against variance; gamma must be tuned.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet FEMNIST PACS ImageNet_1K
Benchmarks: CIFAR-100 Tiny-ImageNet PACS
Metrics: average accuracy variance of client accuracy lowest 10% client accuracy lowest 20% client accuracy
Baselines: FedAvg FedMeta q-FFL CoPreFL-SGD Centralized pre-training SCAFFOLD FedDyn PerFedAvg Random initialization FedProx
Builds on: MAML FedAvg q-FFL FedMeta
Closest papers by meaning
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.88)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.87)
- Convergence Analysis of Split Federated Learning on Heterogeneous Data (NeurIPS 2024 · similarity 0.87)
- FLUX: Efficient Descriptor-Driven Clustered Federated Learning under Arbitrary Distribution Shifts (NeurIPS 2025 · similarity 0.86)
- BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach (ICML 2025 · similarity 0.86)
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.