Research map / Reinforcement learning / Visual and representation RL
Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes reward-respecting subtasks of feature attainment (using the original reward plus a bonus for stopping in high-value feature states) to discover options whose learned models are more useful for planning via the STOMP progression.
Problem
Option discovery for model-based RL is typically driven by subtasks that ignore the main-task reward, so the learned option models are rarely useful for planning; the paper asks how to discover options whose models help planning.
Approach
Define subtasks whose cumulant is the original reward and whose stopping value is the estimated main-task value with an optimistic bonus weight on a chosen feature, so options trade off reward, value of stopping state, and a stopping bonus; learn options, expectation models and plan with approximate value iteration using off-policy TD(λ)-style updates.
Key results
- In the two-room gridworld, planning with the reward-respecting option model was much faster than with the shortest-path option model, which was slower than actions alone.
- In the four-room stochastic gridworld, planning with reward-respecting option models was fastest, followed by shortest-path options, eigenoptions, and option-critic options.
- Bonus weight of 100 yields shortest-path-like options, while bonus weight 1 yields roundabout reward-avoiding options.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a reward-respecting option instead of a shortest-path option to the hallway | Planning in the two-room gridworld · Two-room gridworld | planning efficiency | Helped |
| Use a shortest-path option instead of primitive actions alone | Planning in the two-room gridworld · Two-room gridworld | planning efficiency | Hurt |
| Use reward-respecting options instead of shortest-path, eigenoptions, or option-critic options | Planning in the four-room gridworld · Four-room gridworld | planning speed | Helped |
| Use options discovered by option-critic instead of primitive actions alone | Planning in the four-room gridworld · Four-room gridworld | estimated start-state value during planning | Mixed |
| Increase the hallway bonus weight from 1 to 10 (ablation) | Reward-respecting hallway option and subsequent planning · Two-room gridworld | planning efficiency | Helped |
| Increase the hallway bonus weight from 1 to 100 (ablation) | Reward-respecting hallway option and subsequent planning · Two-room gridworld | planning efficiency | Helped |
| Decrease the hallway bonus weight from 1 to 0.1 (ablation) | Reward-respecting hallway option and subsequent planning · Two-room gridworld | planning efficiency | Mixed |
| Set the hallway bonus weight to 100, producing a shortest-path-like option (ablation) | Reward-respecting hallway option · Two-room gridworld | path length to hallway | No clear effect |
Surprising findings
- Planning with the shortest-path option model was slower than planning with primitive actions alone in the two-room gridworld.
- Option-critic options performed poorly in planning despite being reward-respecting, attributed to on-policy learning giving inaccurate values outside visited states.
Limitations
- Experiments are limited to small tabular gridworlds with one-hot features.
- Sequential STOMP steps are used; simultaneous operation and nonlinear function approximation are not evaluated.
- Reward-respecting subtasks are only one small subset of possible subtasks; the paper focuses on feature-attainment subtasks.
Experimental setup
Benchmarks: two-room gridworld four-room stochastic gridworld
Metrics: estimated value of start state during planning RMSE of option value function RMSE of model Monte Carlo estimate of policy value
Baselines: eigenoptions option-critic shortest-path options primitive actions only
Builds on: options framework (Sutton, Precup & Singh 1999) general value functions (Sutton et al. 2011) eigenoptions (Machado et al. 2017, 2018) option-critic (Bacon et al. 2017) expectation models (Wan et al. 2019)
Closest papers by meaning
- Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.89)
- Reinforcement Learning with Lookahead Information (NeurIPS 2024 · similarity 0.88)
- Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.87)
- Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- OptionZero: Planning with Learned Options (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.