Research map / Bandits and online learning / RL theory and MDPs
Automatic Reward Shaping from Confounded Offline Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives causal Bellman optimality upper bounds on optimal state values from confounded offline data and uses them as potentials for PBRS in a model-free Q-UCB learner, proving a gap-dependent regret improvement.
Problem
Designing potential functions for potential-based reward shaping is hard when offline data is confounded by unobserved variables, since standard off-policy value estimates are biased and not identifiable.
Approach
Compute optimistic causal upper bounds on optimal interventional state values from confounded offline datasets via a Causal Bellman Optimality Equation (partial identification), use them as state potentials in PBRS, and plug them into a modified Q-UCB (Q-UCB Shaping) with clipped potential-dependent bonuses.
Key results
- On windy LavaCross easy and hard, only the causal upper-bound shaping converges to the safe optimal policy, while min/max/avg behavioral-value shaping slows learning.
- Optimal-policy ratio on LavaCross (maze) is 0.90 for Shaping + Causal Bound versus 0.24-0.33 for the naive behavioral-value shaping variants and 0.03 for no shaping.
- The paper proves a gap-dependent regret bound for Q-UCB Shaping with H^5 dependence on pseudo-suboptimal pairs versus H^6 otherwise, claimed to be the first gap-dependent regret bound for PBRS in model-free online learning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use causal upper-bound potentials instead of the strongest reported baseline | Q-UCB shaping · Windy Empty World | optimal ratio: 0.88 -> 0.76 | Hurt |
| Use causal upper-bound potentials instead of Shaping + Max. Val. | Q-UCB shaping · LavaCross (easy) | optimal ratio: 0.52 -> 0.70 | Helped |
| Use causal upper-bound potentials instead of BCQ | LavaCross (hard) · Q-UCB shaping | optimal ratio: 0.67 -> 0.81 | Helped |
| Use causal upper-bound potentials instead of BCQ | LavaCross (maze) · Q-UCB shaping | optimal ratio: 0.50 -> 0.90 | Helped |
| Use causal upper-bound potentials rather than confounded behavioral values | Q-UCB shaping · Windy LavaCross (easy and hard) | learning efficiency | Helped |
| Use causal upper-bound potentials rather than confounded behavioral values | Q-UCB shaping · Windy LavaCross (maze) | learning efficiency | Helped |
| Use causal-bound shaping rather than no shaping | Q-UCB · Windy Empty World | cumulative regret | No clear effect |
Surprising findings
- Naively using confounded offline state values as potentials can impair training efficiency relative to no shaping.
- Shaping with BCQ-learned values reaches 0.88 optimal ratio on Windy Empty World but drops to 0.03 on LavaCross maze.
Limitations
- Results are limited to small tabular grid worlds with deterministic rewards and three seeds.
- Upper bound requires a known reward upper bound b and visited state-action pairs in the offline data; unvisited states use a loose default bound (H-h)b.
Experimental setup
Datasets: Windy Empty World Windy LavaCross (easy) Windy LavaCross (hard) Windy LavaCross (maze)
Benchmarks: Windy Empty World Windy LavaCross (easy) Windy LavaCross (hard) Windy LavaCross (maze)
Metrics: cumulative regret optimal ratio
Baselines: No Shaping Q-UCB Shaping + Min Beh. Value Shaping + Max Beh. Value Shaping + Avg Beh. Value BCQ Shaping + BCQ
Builds on: Potential-Based Reward Shaping Q-UCB Bellman optimality equation partial identification bounds for confounded MDPs
Closest papers by meaning
- ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.90)
- BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping (ICLR 2025 · similarity 0.89)
- Bootstrapped Reward Shaping (AAAI 2025 · similarity 0.89)
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Information-Directed Offline-to-Online Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.