Research map / Reinforcement learning / Visual and representation RL

Potential-Based Reward Shaping for Intrinsic Motivation (Student Abstract)

AAAI 2024 · Reinforcement learning › Visual and representation RL · approach: Reward shaping for sparse rewards · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends potential-based reward shaping to arbitrary episodic shaping functions and proposes PBIM, which converts intrinsic motivation rewards into a potential-based form that preserves the optimal policy set, tested on MiniGrid DoorKey 8x8.

Problem

Intrinsic motivation reward shaping can change the set of optimal policies, causing suboptimal or 'hacked' behavior, and prior PBRS guarantees do not cover IM rewards that are trainable functions of many variables.

Approach

Generalizes the PBRS optimality condition to shaping rewards that depend on arbitrary variables in episodic settings, and converts IM rewards into a potential-based form via a terminal-state correction term (plus a mean-adjusted variant) so the optimal policy set is preserved.

Key results

• PBIM converged in 1.67E6 steps versus 2.88E6 for IM without PBIM at alpha=0.02, gamma=0.995 (table 1 setting), with mean episode length 51.8 vs 60.5.
• Without PBIM normalization, IM runs at alpha=0.02 and 0.25 failed to converge to the efficient policy (mean episode length ~635 steps), while PBIM converged (~52 steps).
• At alpha=0.005, gamma=0.99 PBIM converged faster than IM without PBIM (T=1.67E6 vs 1.46E6 is slower, but final episode length 36.4 vs 37.3 is better), so the method shows a speed/safety trade-off in the worst case.

Measured effects reported in the paper

Surprising findings

• In the alpha=0.005, gamma=0.99 setting the normalized PBIM was not fastest to converge, though it reached a more efficient final policy than IM without PBIM.

Limitations

• Theory applies only under Assumption 1 that the shaping reward is constant with respect to actions at later times, which excludes some IM methods.
• Empirical validation limited to one small environment (MiniGrid DoorKey 8x8) with a tabular count-based reward.
• In the worst-case hyperparameter setting PBIM converged more slowly than IM without PBIM.

Experimental setup

Datasets: MiniGrid DoorKey 8x8

Benchmarks: MiniGrid DoorKey 8x8

Metrics: time to convergence mean steps per episode after convergence std of steps after convergence

Baselines: IM without PBIM PBIM without normalization no intrinsic motivation

Builds on: potential-based reward shaping (Ng, Harada, Russell 1999) Wiewiora et al. 2003 PBRS Burda et al. 2018 curiosity

Closest papers by meaning

• Action-Dependent Optimality-Preserving Reward Shaping (ICML 2025 · similarity 0.90)
• BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping (ICLR 2025 · similarity 0.89)
• Bootstrapped Reward Shaping (AAAI 2025 · similarity 0.88)
• Automatic Reward Shaping from Confounded Offline Data (ICML 2025 · similarity 0.88)
• Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.87)
• ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.86)