Research map / Reinforcement learning / Visual and representation RL
Intelligent Switching for Reset-Free RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RISC, a reset-free RL method that alternates forward and backward (reset) controllers and switches early based on a learned success-critic competency estimate, with timeout-nonterminal bootstrapping, reaching state-of-the-art on several EARL benchmark tasks.
Problem
Reinforcement learning without environment resets is hard because agents lose the ability to revisit initial states and exit bad regions, and existing reset-free methods use fixed switching times and inconsistent bootstrapping at truncated trajectory ends.
Approach
Switch between forward and backward controllers early when a learned success critic indicates the agent is already competent at its current goal, with probability proportional to competency and increasing with trajectory length, while bootstrapping timeout-truncated last states to keep value targets consistent.
Key results
- RISC achieves 1.0 on Tabletop Manipulation and Sawyer Door and 0.42 on Sawyer Peg, the best reset-free results in those tasks (Sawyer Peg tied in range with MEDAL 0.40).
- On Minitaur RISC reaches -321.74 return versus -186.30 for R3L and -986.34 for FBRL.
- Ablations show removing timeout-nonterminal bootstrapping causes a significant performance drop, while removing early switching causes a modest drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add timeout-nonterminal bootstrapping (ablation) | RISC · EARL benchmark | performance | Helped |
| Add early switching based on the success critic (ablation) | RISC · EARL benchmark | performance | Helped |
| Add timeout-nonterminal bootstrapping and early switching (ablation) | RISC · EARL benchmark | performance | Helped |
| Keep the trajectory-length and switching-frequency modulations (ablation) | RISC · EARL benchmark | average return | Helped |
| Use intelligent switching instead of a reverse curriculum | 4 Rooms · 4 Rooms | experience usefulness | Helped |
| Use RISC instead of other reset-free methods | EARL benchmark · EARL benchmark aggregate | IQM performance | Helped |
| Use RISC instead of reset-free baselines | RISC · Sawyer Door | average return: 1.00 ± 0.00 | No clear effect |
| Use RISC instead of reset-free baselines | RISC · Sawyer Peg | average return: 0.42 ± 0.07 vs 0.40 ± 0.16 | Helped |
| Use RISC instead of reset-free baselines | RISC · Minitaur | average return: -321.74 ± 53.73 vs -186.30 ± 34.79 | Mixed |
Surprising findings
- Oracle episodic RL scored lower than RISC on Tabletop Manipulation (0.80 vs 1.0), and RISC matched or exceeded the episodic oracle on Sawyer Door and Sawyer Peg.
Limitations
- May be unsuitable for environments with irreversible states where the agent could get stuck.
- Does not leverage demonstrations to guide exploration intelligently; uses demonstrations only to seed the replay buffer.
- Tabletop and Sawyer Door appear near saturation on EARL.
Experimental setup
Datasets: EARL benchmark demonstrations (Tabletop Manipulation, Sawyer Door, Sawyer Peg) 4 rooms minigrid
Benchmarks: EARL Tabletop Manipulation EARL Sawyer Door EARL Sawyer Peg EARL Minitaur 4 rooms gridworld
Metrics: average return interquartile mean mean normalized score AUC probability of improvement
Baselines: FBRL R3L VapRL MEDAL Naive RL Episodic RL
Builds on: Forward-Backward RL SAC DQN MEDAL EARL benchmark
Closest papers by meaning
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.84)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.84)
- Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.84)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.83)
- A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.