Research map / Reinforcement learning / Offline RL
Federated Ensemble-Directed Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FEDORA, a federated offline RL algorithm that uses entropy-regularized (softmax) ensemble weighting over client policies and critics, optimistic federated critic targets, and proximal regularization with decaying local-data influence.
Problem
Federated offline RL, where clients hold small offline datasets collected by different unknown behavior policies of heterogeneous quality and cannot share data; naive federation of offline RL actors and critics (e.g., FedAvg) can yield policies worse than individual clients.
Approach
Weight client policies and critics with entropy-regularized softmax weights based on each client's estimated performance (scaled by dataset size), use a federated critic to form optimistic targets for local critic updates, regularize local actor updates toward both local data and the federated policy, and decay the local-data loss when the federated policy is estimated to outperform the local upda
Key results
- On simulated MuJoCo tasks with 50 clients (25 expert, 25 medium), FEDORA outperforms all federated baselines and matches or exceeds centralized TD3-BC on pooled data.
- Naive federated baselines (Fed-A, Fed-AC) perform worse than individual offline RL on Hopper in the motivating experiment.
- In the real-world TurtleBot experiment, only FEDORA successfully reaches the target while avoiding the obstacle.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FEDORA's ensemble-directed policy and critic federation instead of standard federated baselines | Federated offline RL · Hopper, HalfCheetah, Walker2D | cumulative episodic reward | Helped |
| Train on pooled heterogeneous client data rather than using FEDORA | Federated offline RL · Hopper | cumulative episodic reward | Hurt |
| Use priority-weighted actor federation (ablation) | FEDORA · Hopper | server performance | Helped |
| Use an optimistic critic (ablation) | FEDORA · Hopper | server performance | Helped |
| Add proximal policy regularization with respect to the federated policy (ablation) | FEDORA · Hopper | server performance | Helped |
| Decay the influence of local data (ablation) | FEDORA · Hopper | server performance | Helped |
| Decay the influence of local data (ablation) | FEDORA · Walker2D | performance | Helped |
| Use average dataset reward for federation weights instead of policy performance (ablation) | FEDORA · Hopper-v2 | cumulative episodic reward | Hurt |
| Use normalized average dataset reward for federation weights instead of policy performance (ablation) | FEDORA · Hopper-v2 | cumulative episodic reward | Hurt |
| Increase the proportion of expert-data clients (ablation) | FEDORA · Hopper | cumulative episodic reward | Mixed |
| Vary the decay parameter δ (ablation) | FEDORA · Hopper-v2 | cumulative episodic reward | No clear effect |
| Add data rebalancing to pooled-data TD3-BC | TD3-BC · Hopper-v2 | performance | Helped |
Surprising findings
- Naively federating offline RL actors and critics with FedAvg yields federated policies worse than individual clients' offline RL.
- Centralized offline RL over pooled data from mixed-quality behavior policies can be detrimental compared with the proposed federated method.
Limitations
- Assumes all clients share the same MDP (transition and reward models); heterogeneous environments not addressed.
- Personalization in federated offline RL is left for future work.
- Hyperparameters beta and delta are fixed by hand (beta=0.1, delta=0.995) and sensitivity is only partially explored.
Experimental setup
Datasets: D4RL hopper-expert-v2 D4RL hopper-medium-v2 D4RL halfcheetah D4RL walker2d NeoRL CityLearn High/Low TurtleBot real-world demonstrations
Benchmarks: MuJoCo Hopper MuJoCo HalfCheetah MuJoCo Walker2D CityLearn TurtleBot real-world navigation
Metrics: episodic reward of federated/server policy
Baselines: Fed-A Fed-AC Fed-AC-Prox HDAFL Centralized TD3-BC CQL IQL TD3-BC with data rebalancing
Builds on: TD3-BC FedAvg FedProx CQL Flower
Closest papers by meaning
- Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices (ICML 2024 · similarity 0.91)
- Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments (ICML 2024 · similarity 0.88)
- Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis (ICLR 2025 · similarity 0.87)
- Single-Loop Federated Actor-Critic across Heterogeneous Environments (AAAI 2025 · similarity 0.87)
- Federated Q-Learning: Linear Regret Speedup with Low Communication Cost (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.