Research map / Reinforcement learning / MDPs and planning
Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes federated natural policy gradient (FedNPG) and federated natural actor-critic (FedNAC) methods for fully decentralized multi-task RL, using gossip mixing of log-policies and gradient tracking of the global Q-function, with non-asymptotic convergence guarantees nearly dimension-free in state-action space size.
Problem
Learning a single globally optimal softmax policy across agents with private per-task rewards and a shared transition kernel, where agents only communicate with graph neighbors and no reward sharing is allowed; prior work lacks non-asymptotic guarantees for decentralized multi-task policy optimizati
Approach
Federated NPG under softmax parameterization mixes neighbors' log-policies via a gossip matrix while tracking the global Q-function with dynamic average consensus (Q-tracking), giving multiplicative policy updates; the entropy-regularized variant yields linear convergence, and a federated actor-critic variant extends this to function approximation with on-policy samples.
Key results
- Exact FedNPG achieves O(1/T) average-iterate rate in the server-client case (sigma=0) and O(1/T^{2/3}) in general networks, with dependence on |A| only through log|A|.
- Entropy-regularized FedNPG converges linearly at rate max{1-tau*eta/2, (3+sigma)/4} to the regularized optimal policy.
- FedNAC sample complexity is O(1/eps^{7/2}) per agent in the decentralized setting and O(1/eps^3) in the server-client setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add entropy regularization to FedNPG | FedNPG for multi-task RL · Regularized versus unregularized multi-task RL | convergence rate | Helped |
| Use a fully connected communication network ($\sigma=0$) | FedNPG · Unregularized multi-task RL | iteration complexity: $\mathcal{O}(1/T)$ | Helped |
| Use a fully connected communication network ($\sigma=0$) | FedNPG with entropy regularization · Regularized multi-task RL | iteration complexity: $\mathcal{O}\Big( \frac{1}{\eta\tau}\log\frac{1}{\varepsilon}\Big)$ | Helped |
| Allow inexact local policy evaluation, with sufficiently small approximation errors | FedNPG · Unregularized multi-task RL | convergence rate | No clear effect |
| Use on-policy sampling and function approximation in place of exact tabular policy evaluation | FedNAC · Function-approximation multi-task RL | sample complexity per agent: $\mathcal{O}\left(\frac{1}{(1-\gamma)^8\varepsilon^3}\right)$; $\mathcal{O}\left(\frac{1}{(1-\gamma)^{45/4}\varepsilon^{ | Mixed |
Limitations
- Main guarantees assume exact policy evaluation in the tabular setting.
- Decentralized rates degrade as the network becomes poorly connected (sigma near 1).
Experimental setup
Metrics: sub-optimality gap iteration complexity sample complexity consensus error
Baselines: NPG Q-NPG
Builds on: natural policy gradient Q-NPG dynamic average consensus gradient tracking
Closest papers by meaning
- Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Single-Loop Federated Actor-Critic across Heterogeneous Environments (AAAI 2025 · similarity 0.91)
- Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments (ICML 2024 · similarity 0.91)
- Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices (ICML 2024 · similarity 0.90)
- Federated Q-Learning: Linear Regret Speedup with Low Communication Cost (ICLR 2024 · similarity 0.89)
- Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.