Research map / Reinforcement learning / Visual and representation RL
SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SLOPE, which turns sparse-reward MBRL into planning on a dense potential landscape (PBRS with a target-network Q as potential) plus an asymmetric quantile-weighted cross-entropy loss for optimistic value learning, evaluated on TD-MPC2 across simulated benchmarks and real-robot tasks.
Problem
Model-based RL under sparse rewards suffers from reward models that collapse to zero and flat planning landscapes, giving MPPI-style planners no gradient to guide exploration toward success.
Approach
Replaces scalar reward regression with potential-based reward shaping using the agent's slow target Q-function as potential, and biases the distributional value learning toward upper quantiles via asymmetric cross-entropy weighting so rare success signals are amplified.
Key results
- On real-world Grasp Cube at 15k steps, SLOPE reaches 65.0% success vs. 35.0% for MoDem, 20.0% for DEMO^3 and 0.0% for SAC.
- On real-world Push Cube, SLOPE reaches 65.0% success vs. 15.0% for MoDem and DEMO^3.
- Across 20 simulation tasks from 4 benchmarks, SLOPE is reported to consistently outperform TD-MPC2, MoDem and DEMO^3 variants under sparse rewards (exact aggregate numbers not given in the extracted text).
Surprising findings
- Disabling the accelerated training strategies causes only a marginal drop, while removing the potential landscape and optimism components severely degrades performance.
- TD-MPC2 with sparse rewards fails across all tasks, and DEMO^3-sparse suffers a notable drop without handcrafted stage priors.
Limitations
- Optimistic potential scaling may encourage high-risk behavior in unvisited hazardous regions; safety is not addressed.
- Hyperparameter sensitivity: performance degrades for eta above 1.0 and for tau above 0.55.
- Real-world evaluation uses only three tasks, with human-verified success signals and 20 trials per method.
Experimental setup
Datasets: ManiSkill3 Meta-World Adroit Robosuite DeepMind Control Suite
Benchmarks: ManiSkill3 Meta-World Adroit Robosuite DeepMind Control Suite real-world Press Button real-world Push Cube real-world Grasp Cube
Metrics: success rate intervention rate
Baselines: TD-MPC2 MoDem DEMO^3 Behavior Cloning SAC DreamSmooth LS-Imagine
Builds on: TD-MPC2 MoDem DEMO^3 Potential-Based Reward Shaping HIL-SERL
Closest papers by meaning
- Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling (ICLR 2025 · similarity 0.88)
- SOMBRL: Scalable and Optimistic Model-Based RL (NeurIPS 2025 · similarity 0.87)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.87)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
- DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing (ICLR 2024 · similarity 0.86)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.