Research map / Reinforcement learning / Visual and representation RL

Action-Dependent Optimality-Preserving Reward Shaping

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: Reward shaping for sparse rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADOPS, an action-dependent optimality-preserving reward shaping method that adjusts intrinsic rewards using critic estimates so they preserve optimal policies, and shows gains over PBIM, GRM, PIES and RND on Montezuma's Revenge.

Problem

Potential-based reward shaping methods (PBIM, GRM, PIES) that preserve optimal policies under intrinsic motivation fail in complex, exploration-heavy sparse-reward environments with long episodes, and PBRS requires action-independent intrinsic returns.

Approach

Convert any shaping reward into an optimality-preserving form by adding a correction term F2 computed from the agent's critic estimates of extrinsic and intrinsic values, which only changes the intrinsic reward when it would flip action preference relative to extrinsic rewards; this allows action-dependent intrinsic returns while preserving the optimal policy set, without episodic or future-agnost

Key results

• ADOPES (ADOPS fused with PIES-style scaling) is the best-performing method and significantly outperforms RND (p=0.038) in Montezuma's Revenge.
• All ADOPS variants significantly outperform PIES, GRM and PBIM (e.g., ADOPS vs PIES p=4.4e-5).
• PBIM with RND never obtains nonzero extrinsic reward, because end-of-episode penalties grow to about 10^19 on the longest episodes.

Measured effects reported in the paper

Surprising findings

• PBIM immediately saturates action probabilities and never obtains extrinsic reward in Montezuma's Revenge, because its end-of-episode penalty explodes exponentially with episode length.
• PIES performs well early but declines sharply near its halfway point and never recovers, suggesting its early success came from a lower effective intrinsic coefficient.

Limitations

• Theoretical guarantee relies on an assumption that converged policies are stable (no strictly preferable single-action perturbation), which is not verified in practice.
• Practical ADOPS uses critic estimates rather than optimal values, and critic estimates are poor early in training.
• Evaluated only on a single environment (Montezuma's Revenge) with RND as the base intrinsic motivation.

Experimental setup

Datasets: Montezuma's Revenge (ALE/MontezumaRevenge-v4)

Benchmarks: Montezuma's Revenge

Metrics: average extrinsic return of final policy

Baselines: RND PBIM GRM PIES

Builds on: Potential-Based Reward Shaping Potential-Based Intrinsic Motivation Generalized Reward Matching Policy-Invariant Explicit Shaping Random Network Distillation

Closest papers by meaning

• BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping (ICLR 2025 · similarity 0.91)
• Potential-Based Reward Shaping for Intrinsic Motivation (Student Abstract) (AAAI 2024 · similarity 0.90)
• ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.89)
• Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.89)
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.88)