Research map / Federated learning and privacy / Communication-efficient FL
First-Order Federated Bilevel Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MemFBO, a fully single-loop, first-order federated bilevel algorithm based on a Lagrangian reformulation, with convergence guarantees under partial and full client participation and experiments on federated data cleaning and hyper-representation.
Problem
Federated bilevel optimization (FBO) methods compute federated hypergradients with nested sub-loops and second-order Hessian/Jacobian-vector products, causing high computation and memory cost on edge devices.
Approach
Replace the bilevel problem with a single-level Lagrangian minimax surrogate per client, enabling single-loop local SGD updates on x, y, z using only first-order gradients, with global aggregation and updates on the server.
Key results
- On CIFAR10 federated data cleaning at 50% noise, MemFBO reaches 77.94% test accuracy versus 61.68% for FedRep and 63.27% for SimFBO.
- MemFBO achieves O(P^-2/7 T^-2/7) convergence under partial participation and an improved communication complexity of O(eps^-1.5) for full participation versus O(eps^-2) for FedNest, AggITD and FedMBO.
- Memory cost of SimFBO grows with the number of local steps while MemFBO keeps low memory usage, per Figure 4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MemFBO for federated data cleaning instead of the compared federated learning baselines | Federated data cleaning · CIFAR10 with noise rates 0%, 30%, 50%, and 70% | test accuracy: 81.33 vs 88.93 (0%); 78.21 vs 76.27 (30%); 77.94 vs 71.04 (50%); 73.60 vs 60.44 (70%) | Mixed |
| Use MemFBO's first-order updates instead of SimFBO's second-order updates | Federated hyper-representation on MNIST · MNIST with MLP backbones | memory usage | Helped |
| Use MemFBO instead of FedNest, LFedNest, or AggITD | Federated hyper-representation · MNIST with MLP backbones | test accuracy | Helped |
| Use MemFBO instead of SimFBO | Federated hyper-representation · MNIST with MLP backbones | test accuracy | No clear effect |
Surprising findings
- FedAvg and FedProx drop to around 10-17% accuracy at high noise rates, far below personalized baselines.
Limitations
- Requires a sufficiently large Lagrange multiplier, creating a trade-off with client drift and approximation error.
- Theory assumes strong convexity of the lower-level objective and bounded heterogeneity assumptions.
- Experiments are limited to CIFAR10 and MNIST with small CNN/MLP models.
Experimental setup
Datasets: CIFAR10 MNIST
Benchmarks: CIFAR10 federated data cleaning (noisy label rates 0-70%) MNIST non-i.i.d. hyper-representation
Metrics: average test accuracy communication rounds memory cost convergence rate
Baselines: FedRep FedPer LG-Fed FedAvg FedProx SimFBO FedNest LFedNest AggITD FedMBO
Builds on: SimFBO FedNest first-order bilevel Lagrangian methods
Closest papers by meaning
- First-Order Minimax Bilevel Optimization (NeurIPS 2024 · similarity 0.86)
- Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis (ICLR 2025 · similarity 0.86)
- Problem-Parameter-Free Decentralized Bilevel Optimization (NeurIPS 2025 · similarity 0.86)
- Decomposable Submodular Maximization in Federated Setting (ICML 2024 · similarity 0.86)
- Federated Optimization with Doubly Regularized Drift Correction (ICML 2024 · similarity 0.85)
- Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.