Research map / Optimization / Distributed and compressed
CoBo: Collaborative Learning via Bilevel Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoBo, an SGD-type alternating bilevel method that jointly learns personalized client models and pairwise collaboration weights via gradient alignment, with convergence guarantees for cluster-structured clients.
Problem
Collaborative learning across heterogeneous clients where a global model underperforms and identifying which clients to collaborate with is costly and typically requires clustering with predetermined cluster counts.
Approach
Model client selection (binary-relaxed pairwise weights set by gradient inner products at averaged models) and personalized model training as a bilevel problem; solve by alternating projected gradient steps on weights with SGD steps on models, penalizing pairwise distances weighted by learned collaboration.
Key results
- In the 80-client cross-device CIFAR-100 setting, CoBo reaches 79.6% accuracy vs 70.3% for Ditto (about 9.3 points higher, matching the abstract's claim).
- In the 8-client cross-silo setting, CoBo achieves 74.6% accuracy vs 73.5% for Ditto and 75.4% for the oracle.
- On Wiki-40B language fine-tuning with 4 clients, CoBo obtains perplexity 39.28 vs 40.05 for Ditto and 41.26 for local training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CoBo instead of Ditto | personalized federated learning · CIFAR-100 cross-silo (8 clients) | accuracy: 73.5 ± -> 74.6 ± | Helped |
| apply CoBo instead of Ditto | personalized federated learning · CIFAR-100 cross-silo (8 clients) | loss: 1.55 -> 1.08 | Helped |
| apply CoBo instead of Ditto | personalized federated learning · CIFAR-100 cross-device (80 clients) | accuracy: 70.3 ± -> 79.6 ± | Helped |
| apply CoBo instead of Ditto | personalized federated learning · CIFAR-100 cross-device (80 clients) | loss: 1.21 -> 0.97 | Helped |
| apply CoBo instead of Ditto | LLM collaborative fine-tuning · Wiki-40B language modeling (4 clients) | perplexity: 40.05 ± -> 39.28 ± | Helped |
| use constant pair-sampling rate O(1/n) rather than updating all pairs (ablation) | CoBo · CIFAR-100 cross-silo (8 clients), sampling strategies | accuracy: 74.93 -> 73.05 | Hurt |
| use constant pair-sampling rate O(1/n) rather than updating all pairs (ablation) | CoBo · CIFAR-100 cross-silo (8 clients), sampling strategies | loss: 1.278 -> 1.104 | Hurt |
| use time-dependent pair-sampling rate O(1/t) rather than updating all pairs (ablation) | CoBo · CIFAR-100 cross-silo (8 clients), sampling strategies | accuracy: 74.93 -> 73.18 | Hurt |
| use time-dependent pair-sampling rate O(1/t) rather than updating all pairs (ablation) | CoBo · CIFAR-100 cross-silo (8 clients), sampling strategies | loss: 1.278 -> 1.226 | Helped |
| use mixed pair-sampling: O(1/n) initially, then O(1/t) (ablation) | CoBo · CIFAR-100 cross-silo (8 clients), sampling strategies | accuracy: 74.93 -> 74.77 | Hurt |
| use mixed pair-sampling: O(1/n) initially, then O(1/t) (ablation) | CoBo · CIFAR-100 cross-silo (8 clients), sampling strategies | loss: 1.278 -> 1.081 | Helped |
| switch from constant to mixed pair-sampling (ablation) | CoBo · CIFAR-100 cross-silo (8 clients), sampling strategies | accuracy: 73.05 -> 74.77 | Helped |
Surprising findings
- IFCA degenerates to a fully connected collaboration matrix, performing no better than FedAvg in the cross-silo setting.
- The mixed sampling strategy slightly outperformed the non-sampling oracle variant of CoBo in cross-silo accuracy (74.77 vs 74.93 is actually lower; sampling variants are close to the oracle).
Limitations
- Convergence proof only for a simplified setting with full gradients and exact inner minimization, and under cluster assumptions.
- Cross-device 80-client experiments were run only once due to time constraints, so no error bars for those results.
- Oracle baseline may not be optimal since inter-cluster collaboration could help in practice.
Experimental setup
Datasets: CIFAR-100 Wiki-40B
Benchmarks: CIFAR-100 (cross-silo 8 clients) CIFAR-100 (cross-device 80 clients) Wiki-40B (4 languages)
Metrics: accuracy loss perplexity improved-client percentage
Baselines: FedAvg Fine-tuning FedAvg Ditto IFCA Federated clustering (FC) Local training Oracle
Builds on: Ditto FedAvg Federated clustering IFCA
Closest papers by meaning
- SP-CACW: Convergence-Aware Client Weighting for Selfish Personalized Learning (NeurIPS 2026 · similarity 0.85)
- Aequa: Fair Model Rewards in Collaborative Learning via Slimmable Networks (ICML 2025 · similarity 0.84)
- First-Order Federated Bilevel Learning (AAAI 2025 · similarity 0.84)
- A Kernel Perspective on Distillation-based Collaborative Learning (NeurIPS 2024 · similarity 0.83)
- Collaborative Learning with Different Labeling Functions (ICML 2024 · similarity 0.83)
- Co-Dream: Collaborative Dream Synthesis over Decentralized Models (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.