Research map / Reinforcement learning / Visual and representation RL
BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames intrinsic motivation and reward shaping as BAMDP reward shaping, decomposes BAMDP value into value of information and value of opportunity, and proves BAMDP potential-based shaping functions (BAMPFs) preserve Bayes-optimality in meta-RL and approximate optimality in RL when bounded and monotone.
Problem
Pseudo-rewards (intrinsic motivation and reward shaping) can incentivize counterproductive exploration and reward-hacking, and there is no principled framework for when they preserve optimal behavior in RL or meta-RL.
Approach
Model RL as a BAMDP and extend potential-based shaping to BAMDP histories (BAMPFs), decomposing BAMDP value into value of information and value of opportunity to classify pseudo-rewards and guarantee optimality preservation.
Key results
- In Bernoulli bandit meta-RL, the 1st Winner Pulls BAMPF helps A2C learn exploitative agents faster while still converging to optimal, whereas the unshaped pull-count pseudo-reward converges to an over-exploiting agent with suboptimal regret.
- In Mountain Car with PPO, the max-displacement BAMPF reaches the goal sooner and avoids the reward-hacking policy that the displacement reward converges to.
- BAMPF shaping in a 1-state, 100-lever DQN setting tries more levers than no shaping yet still converges to the optimal correct-lever policy, while entropy bonus causes oscillation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the 1st Winner Pulls BAMPF | A2C meta-learning an RNN-based agent · Bernoulli Bandits | regret | Helped |
| Use 1st Winner Pulls directly as a pseudo-reward instead of as a BAMPF | A2C meta-learning an RNN-based agent · Bernoulli Bandits | regret | Hurt |
| Add a bounded monotone BAMPF based on the count of unique levers tried | DQN · 1-state, 100-lever MDP | levers tried | Helped |
| Add a bounded monotone BAMPF based on the count of unique levers tried | DQN · 1-state, 100-lever MDP | correct lever pull fraction | No clear effect |
| Add an entropy bonus based on the entropy of the last 10 lever pulls | DQN · 1-state, 100-lever MDP | true return | Hurt |
| Add Max Displacement BAMPF using exponentially smoothed maximum displacement | PPO · Mountain Car | episodic extrinsic return | Helped |
| Use current displacement as a reward instead of a history-based potential | PPO · Mountain Car | episodic extrinsic return | Hurt |
| Use current displacement as a classic PBS potential | PPO · Mountain Car | learning progress | No clear effect |
Surprising findings
- Using the 1st winner pull count directly as a pseudo-reward causes meta-learning to converge to an over-exploiting agent, while the BAMPF version does not.
- Converting displacement to classic potential-based shaping preserves optimality but does not help learning in Mountain Car.
Limitations
- Converting pseudo-rewards to BAMPF form is described as a manual process.
- RL-setting optimality guarantee is only approximate and requires bounded monotone potentials; non-monotone terms need decomposition into a BAMPF plus a PBSF.
- Experiments are small-scale (bandits, single-state MDP, Mountain Car).
Experimental setup
Datasets: Bernoulli Bandit meta-RL domain Mountain Car single-state lever MDP
Benchmarks: Bernoulli Bandit Mountain Car single-state lever MDP
Metrics: regret average arms tried episodic extrinsic return correct lever pull fraction
Baselines: displacement reward displacement potential-based shaping entropy bonus 1st winner pulls
Builds on: potential-based reward shaping Bayes-adaptive MDP curiosity RND
Closest papers by meaning
- Action-Dependent Optimality-Preserving Reward Shaping (ICML 2025 · similarity 0.91)
- Automatic Reward Shaping from Confounded Offline Data (ICML 2025 · similarity 0.89)
- Potential-Based Reward Shaping for Intrinsic Motivation (Student Abstract) (AAAI 2024 · similarity 0.89)
- Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.89)
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Bootstrapped Reward Shaping (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.