Research map / Reinforcement learning / MDPs and planning
Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes momentum-based federated policy gradient methods (FedSVRPG-M and FedHAPG-M) that converge exactly to a stationary point of the average performance across arbitrarily heterogeneous MDPs, achieving linear speedup with O(eps^{-3/2}/N) sample complexity.
Problem
Federated reinforcement learning where N agents with arbitrarily different MDPs (reward, transition, initial state distribution) must learn a common policy maximizing average performance without sharing trajectories.
Approach
Integrates momentum (anchoring to the global model's policy-gradient estimate) into federated variance-reduced and Hessian-aided policy gradient with importance sampling, so local multi-step updates converge exactly to a stationary point regardless of heterogeneity magnitude, with constant local step-sizes and one trajectory per local iteration.</究>
Key results
- In tabular random MDPs, FedSVRPG-M with beta=0.1 reaches about 7.96-8.01 average performance across kappa in [0,1.0], versus 6.94-6.97 for the beta=1 baseline (PAvg).
- Theory: both algorithms achieve O(eps^{-3/2}/N) per-agent sample complexity, a linear speedup in N, improving on the O(eps^{-5/3}/N^{2/3}) of prior FRL policy-gradient work.
- On CartPole and HalfCheetah, FedSVRPG-M and FedHAPG-M with tuned beta outperform the PAvg baseline; FedHAPG-M shows smaller variance than FedSVRPG-M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use momentum coefficient β = 0.1 instead of β = 1.0 (ablation) | FedSVRPG-M · Random MDPs, κ = 0 | average performance: 6.965_{\pm 0.07} -> 8.013_{\pm 0.07} | Helped |
| use momentum coefficient β = 0.1 instead of β = 1.0 (ablation) | FedSVRPG-M · Random MDPs, κ = 0.2 | average performance: 6.951 _{\pm 0.06 } -> 7.957_{\pm 0.07} | Helped |
| use momentum coefficient β = 0.1 instead of β = 1.0 (ablation) | FedSVRPG-M · Random MDPs, κ = 0.4 | average performance: 6.955_{\pm 0.06} -> 7.968_{\pm 0.06} | Helped |
| use momentum coefficient β = 0.1 instead of β = 1.0 (ablation) | FedSVRPG-M · Random MDPs, κ = 0.6 | average performance: 6.936 _{\pm 0.06 } -> 7.961_{\pm 0.06} | Helped |
| use momentum coefficient β = 0.1 instead of β = 1.0 (ablation) | FedSVRPG-M · Random MDPs, κ = 0.8 | average performance: 6.940_{\pm 0.06} -> 7.964_{\pm 0.07} | Helped |
| use momentum coefficient β = 0.1 instead of β = 1.0 (ablation) | FedSVRPG-M · Random MDPs, κ = 1.0 | average performance: 6.937 _{\pm 0.07} -> 7.981_{\pm 0.06} | Helped |
Limitations
- FedHAPG-M cannot be evaluated in the tabular setting because J_i is not twice differentiable there.
- Convergence analysis relies on bounded policy score, Hessian, and importance-weight variance assumptions (Assumptions 1-3).
Experimental setup
Datasets: CartPole HalfCheetah random MDPs
Benchmarks: CartPole HalfCheetah random MDPs
Metrics: average performance (mean reward) gradient norm
Baselines: PAvg FedKL FedPG-BR FAPI
Builds on: PAvg IS-MBPG HAPG FedAvg SVRPG
Closest papers by meaning
- Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning (ICLR 2024 · similarity 0.92)
- Single-Loop Federated Actor-Critic across Heterogeneous Environments (AAAI 2025 · similarity 0.92)
- On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations (ICLR 2025 · similarity 0.91)
- Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices (ICML 2024 · similarity 0.90)
- Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.