Research map / Reinforcement learning / Visual and representation RL
Reward-Aware Proto-Representations in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops theory and algorithms (DP, TD, default features) for the reward-aware default representation (DR) and evaluates it against the successor representation in reward shaping, option discovery, count-based exploration, and transfer.
Problem
The successor representation is reward-agnostic, and the default representation (a reward-aware proto-representation from linearly solvable MDPs) has limited theoretical development, no efficient incremental learning algorithms, no function-approximation extension, and little analysis of its eigenve
Approach
Derives DP and TD learning rules for the default representation, shows it spans the same space as the SR under constant rewards with a closed-form eigenvalue relation, extends it to state-action rewards, and defines default features (DFs) that allow transfer when terminal rewards change without access to dynamics.
Key results
- In reward shaping with low-reward regions, DR-pot performs significantly better than SR-pot, SR-prior, and no shaping over 50 seeds.
- In count-based exploration on RiverSwim, Sarsa+DR reaches about 2,964 (x10^3) return versus 1,206 for Sarsa+SR and 25 for Sarsa.
- Default features enable near-optimal transfer when only terminal rewards change, computing the optimal policy directly without transition dynamics, outperforming successor features with limited policies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the DR norm as the count-based exploration bonus instead of the SR norm | Sarsa exploration · RiverSwim | total undiscounted return: 1,206 (566) -> 2,964 (252) | Helped |
| Use the DR norm as the count-based exploration bonus instead of no representation bonus | Sarsa exploration · RiverSwim | total undiscounted return: 25 (0.8) -> 2,964 (252) | Helped |
| Use the DR norm as the count-based exploration bonus instead of the SR norm | Sarsa exploration · SixArms | total undiscounted return: 1,066 (2,708) -> 3,518 (4,571) | Mixed |
| Use potential-based shaping with the DR top eigenvector instead of SR-based shaping | Q-learning · Environments with low-reward regions | undiscounted return | Helped |
| Use RACE with the DR instead of CEO with the SR | Online eigenoption discovery · Environments with low-reward regions | average reward per time step | Helped |
| Use RACE with the DR instead of CEO with the SR | Online eigenoption discovery · Environments with low-reward regions | state visitation percentage | Hurt |
| Combine RACE rather than CEO with Q-learning | Q-learning with online eigenoption discovery · Environments with low-reward regions | undiscounted return | Helped |
| Use the DR instead of the SR for reward shaping | Q-learning · Environments without low-reward regions | undiscounted return | No clear effect |
Surprising findings
- Sarsa+DR outperforms Sarsa+SR on RiverSwim, despite the DR's aversion to negative rewards.
- When all non-terminal rewards are constant, the DR behaves essentially the same as the SR.
Limitations
- Numerical instability from exponentiating negative rewards in DR computation and eigendecomposition.
- Default features only support changes in terminal state rewards, not non-terminal rewards.
- Experiments are restricted to the tabular case.
Experimental setup
Datasets: grid task four rooms grid room grid maze RiverSwim SixArms
Benchmarks: RiverSwim SixArms four rooms with multiple goals grid environments
Metrics: undiscounted return percentage of states visited average reward per time step cumulative return across terminal reward configurations
Baselines: SR-pot SR-prior no shaping CEO random walk Sarsa successor features Q-learning
Builds on: successor representation linearly solvable MDPs covering eigenoptions successor features Q-learning
Closest papers by meaning
- Proto Successor Measure: Representing the Behavior Space of an RL Agent (ICML 2025 · similarity 0.89)
- A Distributional Analogue to the Successor Representation (ICML 2024 · similarity 0.88)
- Abstract and Explore: A Novel Behavioral Metric with Cyclic Dynamics in Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Learning Successor Features the Simple Way (NeurIPS 2024 · similarity 0.88)
- Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024 · similarity 0.87)
- Distributional Successor Features Enable Zero-Shot Policy Optimization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.