Research map / Reinforcement learning / MDPs and planning
Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedLCB-Q, a federated offline Q-learning algorithm with rescaled learning rates, importance averaging, and a global pessimistic penalty, achieving linear speedup under collaborative single-policy coverage and O(H) communication rounds.
Problem
Offline RL across multiple agents with local datasets that cannot be shared; existing federated Q-learning lacks pessimism and requires full state-action coverage, while single-agent offline RL needs the behavior policy to cover the optimal policy.
Approach
Incorporates pessimism into federated Q-learning via learning rates that decay with a global counter during aggregation and with local counters during local updates, importance-weighted server averaging, and a global Hoeffding-style penalty computed from aggregated visitation counts; only collaborative (average) single-policy concentrability is needed.
Key results
- FedLCB-Q finds an epsilon-optimal policy once per-agent samples T exceed roughly H^7 S C / (M eps^2), giving linear speedup in number of agents M under collaborative coverage.
- With exponential synchronization (gamma = 2/H), the number of communication rounds is O(H) up to log factors, nearly independent of S, A, M and K.
- For M=1 the bound nearly matches single-agent pessimistic Q-learning up to a factor of H, and is within H^3 of the minimax lower bound for centralized offline RL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add collaborative coverage across M agents (average concentrability) with linear speedup in agents | FedLCB-Q · offline tabular episodic MDP theory | sample complexity | Helped |
| Exponential synchronization schedule (gamma=2/H) instead of periodic | FedLCB-Q · communication rounds | number of synchronizations | Helped |
| Importance averaging and learning rate rescaling in local updates (vs periodic ones) | FedLCB-Q · theory: sample complexity gap to centralized lower bound | polynomial factor in H | Mixed |
Surprising findings
- Single-agent offline RL requires individual coverage of the optimal policy, whereas federated collaboration needs only collective coverage, so learning becomes possible in cases infeasible for any single agent.
Limitations
- Horizon dependence is not minimax-optimal (gap of H factors relative to lower bound)
- Analysis restricted to episodic tabular MDPs
- Assumes all agents share a common MDP without perturbations
Experimental setup
Metrics: sample complexity communication rounds
Baselines: LCB-Q LCB-Q-Adv VI-LCB PEVI-Adv FedQ-Avg
Builds on: Q-learning LCB-Q (pessimistic Q-learning) FedQ-Avg
Closest papers by meaning
- Federated Q-Learning: Linear Regret Speedup with Low Communication Cost (ICLR 2024 · similarity 0.92)
- Federated $Q$-Learning with Reference-Advantage Decomposition: Almost Optimal Regret and Logarithmic Communication Cost (ICLR 2025 · similarity 0.91)
- Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Federated Ensemble-Directed Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.