Research map / Reinforcement learning / MDPs and planning
Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AFedPG, an asynchronous federated policy gradient method with a delay-adaptive lookahead and normalized server updates, proving global and stationary-point convergence and linear sample-complexity speedup with N agents, plus MuJoCo experiments.
Problem
Federated reinforcement learning with policy gradients suffers from stragglers under synchronous aggregation, and asynchronous updates introduce lagged policies because RL agents collect fresh on-policy data from the current model, which prior async FL analyses (supervised, fixed datasets) do not ha
Approach
Agents sample at a delay-adaptive lookahead point computed from the last two global iterates, so second-order Hessian correction terms cancel in the error recursion; the server applies normalized, momentum-averaged policy gradients as each agent's result arrives.
Key results
- AFedPG achieves O(epsilon^-2.5/N) per-agent sample complexity for global convergence, a linear speedup over single-agent PG's O(epsilon^-2.5).
- Global time improves from O(t_max/N) for synchronous FedPG to O(harmonic mean of t_i), with AFedPG showing much lower time than synchronous FedPG in all four MuJoCo tasks.
- Vanilla AFedPG without the delay-adaptive lookahead gives unsatisfactory performance in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use asynchronous updates instead of synchronous FedPG | AFedPG · Federated policy-gradient training | global time complexity: \mathcal{O}(\frac{t_{\max}}{N} {\epsilon}^{-2.5}) -> \mathcal{O}(\bar{t}{\epsilon}^{-2.5}) | Helped |
| use federated agents instead of single-agent PG | AFedPG · Global convergence | sample complexity per agent: \mathcal{O}(\epsilon^{-2.5}) -> \mathcal{O}(\frac{{\epsilon}^{-2.5}}{N}) | Helped |
| increase the number of federated agents | AFedPG · Swimmer-v4, Hopper-v4, Walker2D-v4, Humanoid-v4 | reward | Helped |
| use AFedPG instead of synchronous FedPG | AFedPG · Swimmer-v4, Hopper-v4, Walker2D-v4, Humanoid-v4 | global time consumption | Helped |
| use AFedPG instead of A3C | AFedPG · MuJoCo environments | reward | Helped |
| remove the delay-adaptive lookahead technique (ablation) | AFedPG · MuJoCo environments | reward performance | Hurt |
| increase computation heterogeneity ratio | AFedPG · Federated training with heterogeneous agent speeds | speedup over FedPG | Helped |
| increase agent count in AFedPG | AFedPG · Long-run MuJoCo experiments | time to reach optimal reward | Helped |
Surprising findings
- With enough samples, AFedPG reaches similar final reward across different agent counts, with the main gain being speed of convergence.
- The suboptimal PG baseline on Humanoid-v4 was attributed to hyperparameter tuning difficulty, making its average curve lower than the federated variants.
Limitations
- Robustness to adversarial or malicious workers is not addressed in the asynchronous setting.
- Analysis restricted to discounted rewards; average-reward setting left open.
- Second-order (natural PG) methods are not extended to the asynchronous federated setting.
Experimental setup
Datasets: Swimmer-v4 Hopper-v4 Walker2D-v4 Humanoid-v4
Benchmarks: Swimmer-v4 Hopper-v4 Walker2D-v4 Humanoid-v4
Metrics: average trajectory reward global time to given sample count
Baselines: PG (N=1) synchronous FedPG A3C AFedPG without delay-adaptive lookahead (vanilla)
Builds on: REINFORCE FedPG A3C Generalized advantage estimation stable-baselines3
Closest papers by meaning
- Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning (ICLR 2024 · similarity 0.93)
- Single-Loop Federated Actor-Critic across Heterogeneous Environments (AAAI 2025 · similarity 0.90)
- Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments (ICML 2024 · similarity 0.89)
- Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices (ICML 2024 · similarity 0.89)
- Federated Q-Learning: Linear Regret Speedup with Low Communication Cost (ICLR 2024 · similarity 0.89)
- Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.