Research map / Reinforcement learning / Visual and representation RL

Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning

ICLR 2025 · Reinforcement learning › Visual and representation RL · approach: Reward shaping for sparse rewards · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SASR, a self-adaptive reward shaping method that samples shaped rewards from Beta distributions parameterized by KDE/RFF-estimated success and failure state counts, evaluated with SAC on extremely sparse-reward MuJoCo, robotics and Atari tasks.

Problem

Extremely sparse rewards in RL make exploration slow; existing reward shaping methods either overlook state values (intrinsic novelty bonuses) or struggle early in training (value-based heuristics), and count-based shaping is hard in continuous, high-dimensional state spaces.

Approach

Derive shaped rewards from success rates of states computed from historical successful vs. failed trajectories, sampled from Beta distributions whose variance shrinks as data accumulates; KDE with random Fourier features efficiently estimates the success and failure counts in continuous spaces.

Key results

• SASR achieves the highest average episodic return on most of the 14 tasks, e.g. 94.9 on AntStand versus 67.3 for DRND-online and 31.6 for SAC.
• On Montezuma's Revenge, SASR scores 6737.9 versus 6828.5 for DRND-online, so DRND-online is slightly higher there.
• Removing Beta sampling drops AntStand from 94.92 to 54.48 and HumanStand from 79.83 to 9.77.

Measured effects reported in the paper

Surprising findings

• Using a fixed deterministic success rate instead of Beta sampling sharply degrades performance, showing stochastic reward sampling matters for exploration.
• Using a retention rate of 1 (storing all states) causes premature overconfidence and large drops, e.g. RobotPush falls from 167.14 to 2.96.

Limitations

• Sensitive to the retention rate; the method does not account for varying importance of states within a trajectory.
• Designed for sparse-reward settings; extra shaped rewards may be unnecessary in dense-reward settings.
• ReLara outperforms SASR on RobotReach, which the authors attribute to access to action information.

Experimental setup

Datasets: MuJoCo tasks robotic tasks Atari games MountainCar

Benchmarks: AntStand AntFar HumanStand HumanKeep RobotReach RobotSlide RobotPush RobotPickPlace Pitfall Frogger Montezuma's Revenge Solaris

Metrics: average episodic return standard error sample efficiency training time peak memory

Baselines: DRND ReLara GFA-RFE ROSA ExploRS #Explo RND SAC TD3 PPO

Builds on: soft actor-critic random fourier features thompson sampling random network distillation

Closest papers by meaning

• Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.91)
• Catching Two Birds with One Stone: Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation (ICML 2025 · similarity 0.89)
• Bootstrapped Reward Shaping (AAAI 2025 · similarity 0.89)
• ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.88)
• Automatic Reward Shaping from Confounded Offline Data (ICML 2025 · similarity 0.88)
• BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping (ICLR 2025 · similarity 0.88)