Research map / Reinforcement learning / Visual and representation RL
Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CenRA, a multi-task RL framework where a centralized reward agent trained on pooled replay data generates dense knowledge rewards for distributed policy agents, with a similarity/performance-based sampling weight for balancing tasks.
Problem
Sparse-reward multi-task RL, where knowledge transfer across tasks is slow and the balance of knowledge across tasks is hard to maintain; also transfer to unseen tasks.
Approach
Extends the ReLara assistant reward agent into a single centralized reward agent that serves multiple policy agents: it learns from a concatenated replay buffer of all tasks and emits shared knowledge rewards, with an information-synchronization sampling weight combining task similarity (cross-attention-style softmax over inverse centroid similarity) and recent performance.
Key results
- CenRA achieves 0.875 on ML10-sparse and 0.755 on ML50-sparse episodic return, versus 0.823 for the best baseline TD-MPC2 on ML10 and 0.672 for PiCor on ML50.
- On unseen tasks, CenRA w/o learning reaches 0.887 on ML10-sparse and 0.894 on 2DMaze, outperforming ReLara (0.702 and 0.759).
- Removing both sampling weights drops 2DMaze average performance and raises return variance (1.235 vs 0.021 x10^-2 for full CenRA on 2DMaze).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CenRA instead of the strongest listed baseline, PiCor | CenRA · ML10-sparse | episodic return: 0.865 $\pm$ 0.230 -> 0.875 $\pm$ 0.121 | Helped |
| Use CenRA instead of the strongest listed baseline, PiCor | CenRA · ML50-sparse | episodic return: 0.672 $\pm$ 0.123 -> 0.755 $\pm$ 0.034 | Helped |
| Use CenRA instead of the strongest listed baseline, TD-MPC2 | CenRA · 2DMaze | episodic return: 0.884 $\pm$ 0.046 -> 0.913 $\pm$ 0.023 | Helped |
| Use CenRA instead of the strongest listed baseline, TD-MPC2 | CenRA · 3DPickup | episodic return: 0.712 $\pm$ 0.051 -> 0.880 $\pm$ 0.060 | Helped |
| Use CenRA instead of the strongest listed baseline, TD-MPC2 | CenRA · MujocoCar | episodic return: 505.341 $\pm$ 0.712 -> 514.875 $\pm$ 0.675 | Helped |
| Transfer the pretrained CRA and continue optimizing it on the new task rather than train a new ReLara reward agent | CenRA · ML10-sparse new task | episodic return: 0.702 $\pm$ 0.086 -> 0.902 $\pm$ 0.021 | Helped |
| Transfer the pretrained CRA and continue optimizing it on the new task rather than train a new ReLara reward agent | CenRA · ML50-sparse new task | episodic return: 0.612 $\pm$ 0.012 -> 0.824 $\pm$ 0.012 | Helped |
| Transfer the pretrained CRA and continue optimizing it on the new task rather than train a new ReLara reward agent | CenRA · 2DMaze new task | episodic return: 0.759 $\pm$ 0.056 -> 0.952 $\pm$ 0.010 | Helped |
| Transfer the pretrained CRA and continue optimizing it on the new task rather than train a new ReLara reward agent | CenRA · 3DPickup new task | episodic return: 0.263 $\pm$ 0.002 -> 0.963 $\pm$ 0.002 | Helped |
| Transfer the pretrained CRA and continue optimizing it on the new task rather than train a new ReLara reward agent | CenRA · MujocoCar new task | episodic return: 224.648 $\pm$ 0.492 -> 532.080 $\pm$ 1.610 | Helped |
| Set alpha to 0.25 rather than 0.5 (ablation) | CenRA · 2DMaze, four task returns | episodic return: 0.893 $\pm$ 0.033 -> 0.889 $\pm$ 0.031; 0.908 $\pm$ 0.022 -> 0.901 $\pm$ 0.025; 0.924 $\pm$ 0.020 -> 0.915 $\pm$ 0.023; | Mixed |
| Set alpha to 0.75 rather than 0.5 (ablation) | CenRA · 2DMaze, four task returns | episodic return: 0.893 $\pm$ 0.033 -> 0.891 $\pm$ 0.030; 0.908 $\pm$ 0.022 -> 0.905 $\pm$ 0.024; 0.924 $\pm$ 0.020 -> 0.918 $\pm$ 0.021; | Mixed |
Surprising findings
- Removing the similarity weight alone (alpha=0) gives the highest MujocoCar variance-reduction ablation result on some targets, and the performance weight has a larger impact than the similarity weight, per the authors.
- Full CenRA does not always achieve the lowest variance across ablations, though it consistently outperforms the other ablations in overall performance.
Limitations
- Requires consistent state and action dimensions across tasks.
- The fixed trade-off between similarity and performance weights may not be ideal.
- The performance weight may limit the performance ceiling of high-performing tasks.
Experimental setup
Datasets: Meta-World ML10-sparse Meta-World ML50-sparse 2DMaze 3DPickup MujocoCar
Benchmarks: Meta-World ML10-sparse Meta-World ML50-sparse 2DMaze 3DPickup MujocoCar
Metrics: episodic return variance across tasks
Baselines: DQN/SAC ReLara TD-MPC2 CMTA PiCor MCAL PaCo SC SoftModule
Builds on: ReLara DQN SAC
Closest papers by meaning
- Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.92)
- Cautiously-Optimistic Knowledge Sharing for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.87)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.