Research map / Reinforcement learning / Visual and representation RL
Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-PREP, which plans optimally over task-specific reward machines with shared symbols and feeds the agent the next desired abstract transition plus reshaped rewards, improving few-shot and zero-shot transfer in grid-navigation contextual MDPs.
Problem
Deep RL agents overfit to training tasks and transfer slowly to new contexts under sparse rewards; the paper targets few-shot and zero-shot transfer in contextual MDPs.
Approach
Represent each context by a reward machine over a shared symbol set, run value iteration on the reward machine to find optimal abstract transitions, and provide the agent with the next desired transition label plus potential-based reshaped rewards, so knowledge of symbols and transitions transfers across tasks.
Key results
- C-PREP with CTL improves time-to-threshold AUC by 22.84% (CM) and 42.31% (EL) over the best baseline in the most complex tasks.
- Jumpstart improves by 11.86% (CM) and 36.5% (EL) over the next best baseline.
- In the PO environment, C-PREP yields over 300% improvement in performance over the next best configuration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add C-PREP desired-transition input and reward shaping to CTL | DQN transfer agent · PD, EL and CM | ttt_auc: 37.58 ± 23.38 -> 21.68 ± 6.55; 37.52 ± 25.46 -> 28.95 ± 24.15 | Helped |
| Add C-PREP desired-transition input and reward shaping to CTL | DQN transfer agent · GN, EL and CM | ttt_auc: 6.21 ± 0.42 -> 6.42 ± 0.44; 7.14 ± 1.41 -> 7.64 ± 1.72 | Hurt |
| Add C-PREP desired-transition input and reward shaping to CTL | DQN transfer agent · PO, ON | ttt_auc: 97.50 ± 5.59 -> 22.54 ± 1.57 | Helped |
| Add the Last Transition Label to CTL+C-PREP (ablation) | C-PREP transfer representation · GN, MP, and PD; six EL/CM context settings | ttt_auc | Helped |
| Add reward shaping to the context representation (ablation) | C-PREP transfer agent · GN, MP, and PD | threshold performance | Helped |
| Remove C-PREP components (ablation) | C-PREP transfer agent · ON | task completion after transfer | Hurt |
| Use C-PREP without CTL context information (ablation) | DQN transfer agent · GN, MP, and PD | threshold performance | Hurt |
| Add LTL to C-PREP without CTL (ablation) | C-PREP transfer representation · PO, ON | transfer performance | No clear effect |
Surprising findings
- Agents without reward machine information transfer negatively (negative transfer ratio) in most configurations, and PCG-based context representations are non-beneficial due to overfitting.
- In the ON environment, C-PREP without the controllable context representation performs best, suggesting the reward machine alone can suffice.
Limitations
- Assumes the reward machine generator function is given as input rather than learned.
- Performance depends on the resolution of generated reward machines; partial-resolution machines cover only a small fraction of tasks.
- Evaluated only on small 6x6 grid environments with DQN.
Experimental setup
Datasets: grid navigation environments (GN, MP, PD, ON) with EL, CM, PO context spaces
Benchmarks: GN MP PD ON
Metrics: time to threshold AUC jumpstart transfer ratio
Baselines: CTL CTL+RS CTL+LTL+RS PCG PCG+RS
Builds on: reward machines DQN potential-based reward shaping contextual MDP
Closest papers by meaning
- Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.88)
- Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.88)
- Reward-Aware Proto-Representations in Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Learning Goal-Conditioned Representations for Language Reward Models (NeurIPS 2024 · similarity 0.86)
- Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024 · similarity 0.86)
- Reward Machines for Deep RL in Noisy and Uncertain Environments (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.