Research map / Reinforcement learning / Policy gradient and actor-critic

Single-Loop Federated Actor-Critic across Heterogeneous Environments

AAAI 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Actor-critic theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Single-loop Federated Actor-Critic (SFAC), a two-level federated actor-critic method for heterogeneous environments, and gives finite-time convergence to a near-stationary point of the mixture-environment objective with linear speedup in agents.

Problem

Federated reinforcement learning where N agents interact with heterogeneous MDPs and must learn a shared policy; theoretical convergence of actor-critic in this federated, Markovian, multi-local-update setting was unknown.

Approach

Single-loop federated actor-critic: federated TD critics (FedC) with heterogeneous local update counts are aggregated each round, and federated policy-gradient actors (FedA) are updated once per outer loop using the aggregated critic; the analysis targets the average MSPBE of the mixture environment so the critic bias cancels in the actor update.

Key results

• SFAC obtains higher average return than A3C on LunarLander and CartPole with federated agents.
• Theory: convergence error asymptotically proportional to environment heterogeneity and vanishes as heterogeneity goes to 0.
• Sample complexity exhibits linear speedup with the number of agents N in the homogeneous case.

Measured effects reported in the paper

Limitations

• Convergence only to a near-stationary point, with error floor proportional to environment heterogeneity
• Experiments are small-scale classic control benchmarks with limited comparison (only A3C)

Experimental setup

Datasets: OpenAI Gym LunarLander OpenAI Gym CartPole

Benchmarks: LunarLander CartPole

Metrics: average return

Baselines: A3C

Builds on: policy gradient theorem federated TD learning federated bilevel optimization

Closest papers by meaning

• Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning (ICLR 2024 · similarity 0.92)
• Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments (ICML 2024 · similarity 0.92)
• Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
• Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis (ICLR 2025 · similarity 0.90)
• On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations (ICLR 2025 · similarity 0.89)
• Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices (ICML 2024 · similarity 0.89)