Research map / Reinforcement learning / Visual and representation RL
Reward Shaping for Reinforcement Learning with An Assistant Reward Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReLara, a dual-agent reward shaping framework where a reward agent learns a dense auxiliary reward (via actor-critic on the policy agent's interaction) while a SAC policy agent optimizes the shaped reward, evaluated on sparse-reward continuous control tasks.
Problem
Sparse and delayed rewards in reinforcement learning make exploration and gradient-based learning hard, and existing reward shaping methods rely on hand-crafted rewards or add extra exploration objectives that can cause local optima.
Approach
A cooperative two-agent setup in which a reward agent (actor-critic over state-action pairs, optimizing only the extrinsic reward) generates suggested rewards added to the environment reward for a SAC policy agent; random initial rewards aid exploration and learned rewards later drive exploitation, without extra objectives.
Key results
- ReLara achieves 28.66 on AntStand, versus 15.93 for SAC and 4.54 for PPO in the reported table.
- ReLara achieves 58.71 on RobotPush where all baselines (ROSA, ExploRS, SAC, TD3, RND, PPO) score at most 0.55.
- On MountainCar ReLara reaches 0.89 versus 0.94 for RND and 0.93 for PPO, so it does not win every task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReLara instead of the strongest listed baseline | ReLara · AntStand | average episodic return: 15.93 ± 0.69 -> 28.66 ± 1.82 | Helped |
| Use ReLara instead of the strongest listed baseline | ReLara · AntSpeed, AntFar, AntVeryFar, WalkerKeep, HumanKeep, HumanStand, RobotPush | average episodic return | Helped |
| Use ReLara instead of the strongest listed baseline | ReLara · RobotReach | average episodic return: 110.44 ± 15.00 -> 103.56 ± 7.18 | Hurt |
| Use ReLara instead of the strongest listed baseline | ReLara · MountainCar | average episodic return: 0.94 ± 0.00 -> 0.89 ± 0.01 | Hurt |
| Include the policy agent's action in the reward agent input, rather than using state alone (ablation) | ReLara · continuous control tasks | episodic return | Helped |
| Include the reward agent during both training halves instead of only the first half (ablation) | ReLara · continuous control tasks | episodic return | Helped |
| Include the reward agent during both training halves instead of only the second half (ablation) | ReLara · continuous control tasks | episodic return | Helped |
| Vary the suggested-reward weight factor β (ablation) | ReLara · continuous control tasks | episodic return | Mixed |
Surprising findings
- Random suggested rewards during the early burn-in phase help exploration, as the reward agent's initial random outputs widen the policy's search.
- On RobotReach and MountainCar, ReLara reaches the optimal policy more slowly than PPO and RND due to the burn-in stage.
Limitations
- Advantages rely on sparse, delayed rewards; may be redundant or harmful in dense-reward tasks.
- Two-agent optimization adds algorithmic complexity.
- No formal convergence proof; convergence is only empirically shown.
Experimental setup
Datasets: MuJoCo arm robot (de Lazcano et al., 2023) physical control (Towers et al., 2023)
Benchmarks: AntStand AntSpeed AntFar AntVeryFar WalkerKeep HumanKeep HumanStand RobotReach RobotPush MountainCar
Metrics: average episodic return standard error sample efficiency convergence speed
Baselines: ROSA ExploRS SAC TD3 RND PPO
Builds on: Soft Actor-Critic TD3 ROSA Random Network Distillation
Closest papers by meaning
- Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.91)
- BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping (ICLR 2025 · similarity 0.89)
- Action-Dependent Optimality-Preserving Reward Shaping (ICML 2025 · similarity 0.89)
- Catching Two Birds with One Stone: Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation (ICML 2025 · similarity 0.88)
- Bootstrapped Reward Shaping (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.