Research map / Reinforcement learning / Visual and representation RL
To the Max: Reinventing Reward in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes max-reward RL, which optimizes the maximum reward in an episode via an auxiliary-variable extended MDP with contractive Bellman operators and policy gradient theorems, and applies it to TD3 and PPO on goal-reaching robotics tasks.
Problem
Reward design for goal-reaching RL is hard: dense surrogate rewards are sensitive and sparse rewards are hard to learn from, and prior max-reward Bellman formulations only work in deterministic settings.
Approach
Define max-reward value functions conditioned on an auxiliary variable y tracking the (inversely discounted) max reward so far, yielding Bellman-like contraction operators valid in stochastic MDPs, plus stochastic and deterministic policy gradient theorems; then plug these into TD3 and PPO.
Key results
- In the chain environment, max-reward Q-value iteration converges to the optimal policy much faster than cumulative Q-iteration when goal transitions are skipped with high probability.
- On Maze with DSP rewards for k=1..4, max-reward TD3 and PPO work for all k while cumulative methods degrade as reward becomes sparser (exact numbers only in figures).
- On Fetch-Slide and Fetch-Push, max-reward TD3 reaches a goal-reaching policy while standard TD3 fails to learn.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize maximum rather than cumulative reward | TD3 and PPO · Maze environments with sparse, dense l2, and DSP rewards | success ratio | Helped |
| Optimize maximum rather than cumulative reward | TD3 · Fetch-Slide and Fetch-Push | success ratio | Helped |
| Use the stochastic max-reward formulation instead of deterministic max-reward TD3 (ablation) | TD3 · Stochastic single-goal Maze, DSP reward k=3 | success ratio | Helped |
| Use deterministic max-reward formulation (ablation) | TD3 · Stochastic single-goal Maze, DSP reward k=3 | success ratio | Hurt |
| Increase DSP reward sparsity by using larger k (ablation) | Standard cumulative TD3 and PPO · Maze environments | success ratio | Hurt |
| Increase DSP reward sparsity by using larger k (ablation) | Max-reward TD3 and PPO · Maze environments | success ratio | No clear effect |
| Use sparse reward instead of dense l2 or DSP reward (ablation) | TD3, PPO, and their max-reward variants · Maze environments | success ratio | Hurt |
| Use dense l2 reward instead of DSP reward (ablation) | TD3, PPO, and their max-reward variants · Maze environments | success ratio | Hurt |
Surprising findings
- Naively replacing summation with max in the standard Bellman update gives only a lower bound on expected max-reward return under stochasticity, so it can select suboptimal policies.
- Deterministic max-reward TD3 is highly inconsistent in a stochastic maze, while the proposed stochastic max-reward TD3 solves it.
Limitations
- Experiments limited to two goal-reaching environments (Maze and Fetch).
- Max-reward agents may be more prone to getting stuck in local optima, as noted by the authors' own conclusions.
- Results reported primarily as learning curves; exact numeric comparisons are not given in the text.
Experimental setup
Datasets: Gymnasium-Robotics Maze environment Gymnasium-Robotics Fetch-Slide Gymnasium-Robotics Fetch-Push
Benchmarks: Maze (single-goal and two-goals) Fetch-Slide Fetch-Push
Metrics: success ratio
Baselines: TD3 PPO deterministic max-reward TD3
Builds on: TD3 PPO Q-learning Gymnasium-Robotics
Closest papers by meaning
- Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification (NeurIPS 2021 · similarity 0.90)
- Oracle-Efficient Reinforcement Learning for Max Value Ensembles (NeurIPS 2024 · similarity 0.90)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.89)
- Global Reinforcement Learning : Beyond Linear and Convex Rewards via Submodular Semi-gradient Methods (ICML 2024 · similarity 0.89)
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.89)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.