Research map / Federated learning and privacy / Personalized and heterogeneous FL
Fake It Till Make It: Federated Learning with Consensus-Oriented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedCOG, which generates complementary inputs from the global and local models at each client and distills global knowledge into local training on that generated data, improving federated learning under data heterogeneity without changing communication.
Problem
Data heterogeneity in federated learning causes client drift and limits global model performance; existing methods only correct models rather than the data itself.
Approach
Generate client-side synthetic inputs by optimizing learnable tensors so they are confidently predicted by the global model but disagree with the local model (JS-style disagreement loss), then train the local model with KL distillation from the global model on these inputs plus standard cross-entropy on real data.
Key results
- FedCOG reaches 57.92% average accuracy across six Fashion-MNIST/CIFAR settings, versus 53.68% for FedAvg and the next best FedAvgM at 55.07%.
- On FLAIR, FedCOG achieves 20.89 C-F1 and 51.35 O-F1, about 14% and 15% relative gains over FedAvg.
- Applied on top of FedProx, FedCOG improves FLAIR O-F1 from 45.71 to 54.57 after one extra round.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedCOG instead of the strongest listed baseline | Federated learning · Fashion-MNIST, CIFAR-10 and CIFAR-100 under NIID-1 and NIID-2 | accuracy (%): 77.02 -> 77.34; 70.26 -> 73.68; 64.36 -> 64.83; 53.51 -> 54.00; 42.25 -> 42.88; 34.27 -> 34.80 | Helped |
| Use FedCOG instead of the strongest listed baseline | Federated learning · FLAIR | C-F1 and O-F1: 20.44 -> 20.89; 48.43 -> 51.35 | Helped |
| Add FedCOG to existing FL methods | FedAvg, FedAvgM, FedProx, SCAFFOLD and MOON · CIFAR-10 and FLAIR | accuracy or F1: 63.34 -> 64.47; 49.30 -> 49.50; 45.71 -> 54.57; 20.11 -> 21.98 | Helped |
| Replace Gaussian noise with task-specific generated complementary data (ablation) | FedCOG · CIFAR-10 NIID-1 and NIID-2 | accuracy: 60.98 -> 64.88; 49.67 -> 53.20 | Helped |
| Add disagreement-based global-local generation rather than generating from the global model alone (ablation) | FedCOG · CIFAR-10 NIID-1 and NIID-2 | accuracy: 64.72 -> 64.88; 51.61 -> 53.20 | Helped |
| Use soft-label distillation instead of hard-label supervision (ablation) | FedCOG local-model training · CIFAR-10 NIID-1 and NIID-2 | accuracy: 63.59 -> 64.88; 51.64 -> 53.20 | Helped |
| Use a learnable generator rather than optimizing input tensors directly (ablation) | FedCOG · CIFAR-10 NIID-2 | accuracy: +1.5% absolute accuracy improvement | Helped |
| Use a learnable generator rather than optimizing input tensors directly (ablation) | FedCOG · CIFAR-10 NIID-1 | accuracy: 0.93% absolute accuracy deterioration | Hurt |
| Increase generation steps from 0 to a moderate number (50 or 100) (ablation) | FedCOG · CIFAR-10 | accuracy: 60.98 -> 65.40; 60.98 -> 65.44 | Helped |
| Increase generation steps from 100 to 500 or 1000 (ablation) | FedCOG · CIFAR-10 | accuracy: 65.44 -> 64.80; 65.44 -> 64.92 | Hurt |
| Tune disagreement and distillation loss weights to their reported default settings (ablation) | FedCOG · CIFAR-10 NIID-1 and NIID-2 | accuracy | Helped |
Surprising findings
- Several model-correction baselines (e.g., SCAFFOLD, FedExP) perform worse than FedAvg in some heterogeneous settings.
- Generating data with pure Gaussian noise instead of optimized inputs yields much lower accuracy (60.98 vs 64.88 on CIFAR-10 NIID-1).
Limitations
- Generation adds per-client computation and memory overhead, and the paper notes that large resource-constrained clients need batch-size workarounds.
- Theoretical analysis assumes all clients share the same generated consensus data, which is an idealized assumption.
- Hyperparameters were not carefully tuned, and the learnable generator variant performed on par rather than better.
Experimental setup
Datasets: Fashion-MNIST CIFAR-10 CIFAR-100 FLAIR
Benchmarks: Fashion-MNIST CIFAR-10 CIFAR-100 FLAIR
Metrics: accuracy C-F1 O-F1 precision recall
Baselines: FedAvg FedAvgM FedProx SCAFFOLD MOON FedSAM VHL FedExP FedDecorr FedReg FedGen
Builds on: FedAvg FedProx MOON DeepInversion data-free knowledge distillation
Closest papers by meaning
- An Aggregation-Free Federated Learning for Tackling Data Heterogeneity (CVPR 2024 · similarity 0.91)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.90)
- FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.89)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.89)
- Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learning (ICML 2024 · similarity 0.89)
- Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.