Research map / Reinforcement learning / Visual and representation RL
RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RICE, a refining scheme that uses a StateMask-style mask network to find critical states of a pre-trained DRL agent, resets training to a mix of default and critical initial states, and adds RND exploration bonus to break training bottlenecks.
Problem
Pre-trained deep RL agents, especially under sparse rewards, get stuck in training bottlenecks/local optima; retraining from scratch is costly, and existing refinement (fine-tuning from critical states or random/JSRL frontiers) overfits or fails to escape local optima.
Approach
Build a mixed initial state distribution from default initial states and explanation-identified critical states (from a simplified StateMask mask network trained with PPO and a blinding bonus), then run PPO with an RND exploration bonus from these states, with a theoretical sub-optimality bound argument.
Key results
- On Hopper RICE reaches 3663.91 vs 3559.44 before refining and 3652.06 for StateMask-R; on CAGE Challenge 2 it reaches -20.02 vs -23.64 before refining.
- Explanation fidelity of the simplified mask network is comparable to StateMask, with an average 16.8% reduction in training time.
- Mixing ratio p of 0.25 or 0.5 performed best; p=0 or p=1 performed poorly, and RND bonus lambda>0 gave noticeable gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace StateMask-R fine-tuning with RICE refining | Pre-trained Hopper agent · Hopper | final reward: 3652.06 (8.63) -> 3663.91 (20.98) | Helped |
| Replace StateMask-R fine-tuning with RICE refining | Pre-trained Walker2d agent · Walker2d | final reward: 3966.96 (3.39) -> 3982.79 (3.15) | Helped |
| Replace PPO fine-tuning with RICE refining | Pre-trained Reacher agent · Reacher | final reward: -3.04 (0.04) -> -2.66 (0.03) | Helped |
| Replace PPO fine-tuning with RICE refining | Pre-trained HalfCheetah agent · HalfCheetah | final reward: 2133.31 (4.11) -> 2138.89 (3.22) | Helped |
| Replace PPO fine-tuning with RICE refining | Pre-trained Selfish Mining agent · Selfish Mining | final reward: 14.93 (0.45) -> 16.56 (0.63) | Helped |
| Replace JSRL with RICE refining | Pre-trained Cage Challenge 2 agent · Cage Challenge 2 | final reward: -22.97 (0.57) -> -20.02 (0.32) | Helped |
| Replace PPO fine-tuning with RICE refining | Pre-trained autonomous driving agent · Auto Driving | final reward: 13.37 (3.10) -> 17.03 (1.65) | Helped |
| Replace StateMask-R fine-tuning with RICE refining | Pre-trained malware mutation agent · Malware Mutation | final reward: 50.13 (8.14) -> 57.53 (8.71) | Helped |
| Use a mixed reset distribution rather than only default initial states or only critical states (ablation) | RICE · Applications evaluated | final reward | Helped |
| Enable exploration with a positive RND bonus (lambda > 0) (ablation) | RICE · Applications evaluated | final reward | Helped |
| Use the proposed explanation instead of random critical-state selection (ablation) | RICE · Applications evaluated | final reward | Helped |
| Use the proposed mask-network explanation instead of StateMask (ablation) | RICE · Applications evaluated | final reward | Helped |
Surprising findings
- Refining from critical states only (StateMask-R) can hurt performance, e.g., Cage Challenge 2 drops to -26.98 and HalfCheetah to 2085.28 versus 2024.09 unrefined for PPO-based fine-tuning baselines in some cases.
- JSRL performed poorly in the selected games, sometimes below PPO fine-tuning.
Limitations
- Requires a warm start: the pre-trained policy must already cover the optimal policy's state distribution; it reduces to RND in cold start.
- Requires a resettable simulator to restore critical states.
- Does not filter critical states by their usefulness for further learning.
Experimental setup
Datasets: MuJoCo Hopper MuJoCo Walker2d MuJoCo Reacher MuJoCo HalfCheetah selfish mining simulator CAGE Challenge 2 MetaDrive Macro-v1 MalConv gym environment
Benchmarks: MuJoCo Hopper MuJoCo Walker2d MuJoCo Reacher MuJoCo HalfCheetah selfish mining CAGE Challenge 2 autonomous driving (MetaDrive) malware mutation (MalConv) sparse MuJoCo Hopper sparse MuJoCo HalfCheetah
Metrics: final reward fidelity score training time
Baselines: PPO fine-tuning JSRL StateMask-R Random explanation SAC fine-tuning
Builds on: StateMask Jump-Start Reinforcement Learning Random Network Distillation Proximal Policy Optimization
Closest papers by meaning
- ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies (NeurIPS 2026 · similarity 0.87)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.87)
- CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systems (AAAI 2024 · similarity 0.87)
- DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.86)
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.