Research map / Federated learning and privacy / Communication-efficient FL
On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PFedRL-Rep, a personalized federated RL framework with a shared server-averaged feature representation and per-agent weights, proves linear speedup in agent count for PFedTD-Rep (TD with linear features) under Markovian noise, and extends it to DQN for control.
Problem
Federated RL with heterogeneous environments learns a single policy across agents, which performs poorly per agent; the convergence of personalized FedRL with shared representations under Markovian noise is unexplored.
Approach
Treat personalized FedRL as N parallel RL tasks with shared structure: agents learn a common feature representation via server averaging plus personalized local weight vectors, analyzed as federated two-timescale stochastic approximation with Markovian noise using a Lyapunov drift argument.
Key results
- PFedTD-Rep converges with rate O(N^{-2/3}(T+2)^{-2/3}), giving a linear speedup in the number of agents N under Markovian noise.
- In a 4x12 CliffWalking setup with 3 agents, agent 1 converges in about 50 episodes under PFedTD-Rep versus more than 150 under TD.
- PFedDQN-Rep with 10 CartPole agents (varying pole lengths) obtains larger reward than non-personalized baselines and reaches maximal return faster than personalized baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use shared representations and personalized local weights instead of independent TD | CliffWalking, agent 1 · CliffWalking | episodes to converge: more than 150 episodes -> 50 episodes | Helped |
| use personalized shared representations instead of a single shared policy | PFedTD-Rep · CliffWalking, heterogeneous agents | value-function accuracy | Helped |
| use shared representations with personalization | PFedDQN-Rep · modified CartPole | reward | Helped |
| use shared representations with personalization | PFedDQN-Rep · modified CartPole | speed to maximal return | Helped |
| use shared representations with personalization | PFedDQN-Rep · new environment | adaptation speed | Helped |
Limitations
- Theory covers only linear representations with TD policy evaluation; extension to Q-learning and DQN control is an open problem.
- Convergence rate O(T^{-2/3}) is slower than single-timescale methods and the best known O(T^{-1}) is not attained.
- Experiments are small-scale (tabular CliffWalking, classic control environments).
Experimental setup
Datasets: CliffWalking CartPole Acrobot
Benchmarks: CliffWalking CartPole Acrobot
Metrics: value estimation accuracy episode reward convergence speed (episodes)
Baselines: TD FedTD DQN FedDQN FedQ-K LFRL PerDQNAvg FedAsynQ-ImAvg
Builds on: FedTD FedQ DQN two-timescale stochastic approximation representation learning in federated learning
Closest papers by meaning
- Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments (ICML 2024 · similarity 0.91)
- Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Single-Loop Federated Actor-Critic across Heterogeneous Environments (AAAI 2025 · similarity 0.89)
- Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis (ICLR 2025 · similarity 0.88)
- Federated Q-Learning: Linear Regret Speedup with Low Communication Cost (ICLR 2024 · similarity 0.88)
- Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.