Research map / Reinforcement learning / Policy gradient and actor-critic
NeoRL: Efficient Exploration for Nonepisodic RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeoRL, an optimism-based model-based RL algorithm for nonepisodic (single-trajectory, no-reset) learning of nonlinear systems, with a regret bound O(Γ_T√T) for GP dynamics and experiments on deep RL control benchmarks.
Problem
Learning control of unknown nonlinear dynamical systems from a single trajectory without resets, minimizing average cost while limiting regret, in continuous state-action spaces.
Approach
Optimistic planning over well-calibrated probabilistic dynamics models (GPs or ensembles/BNNs) in the nonepisodic average-cost setting, with horizon selection based on information gain and MPC (iCEM) with hallucinated controls as a practical implementation.
Key results
- NeoRL converges to the optimal average cost across all environments with sublinear cumulative regret using ~10^3 environment interactions.
- NeoRL achieves the lowest regret among NeMean, NePETS, and NeTS, which fail to converge on MountainCar.
- Regret bound O(Γ_T √T) for nonlinear RKHS dynamics under continuity and bounded-energy assumptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use optimistic exploration with NeoRL instead of the baseline methods | model-based RL · all reported environments | average cost | Helped |
| use optimistic exploration with NeoRL instead of the baseline methods | model-based RL · all reported environments | cumulative regret | Helped |
| use NeoRL rather than NeMean | model-based RL · Pendulum-v1 | average cost | Helped |
| use NeoRL rather than NeMean | model-based RL · MountainCar | task success | Helped |
| use NeoRL rather than NeMean | model-based RL · Reacher and CartPole | stability | Helped |
| use NeoRL rather than NePETS | model-based RL · MountainCar | average cost | Helped |
| use NeoRL rather than NePETS | model-based RL · MountainCar | cumulative regret | Helped |
| use NeoRL rather than NeTS | model-based RL · MountainCar | task success | Helped |
| use NeoRL rather than NeMean, NePETS, or NeTS | model-based RL · CartPoleBalance | cumulative regret | Helped |
| use NeoRL rather than NeMean, NePETS, or NeTS | model-based RL · CartPoleBalance | number of resets | Helped |
| use NeoRL with MPC feedback control | NeoRL · Swimmer, SoftArm, and RaceCar | convergence to optimal average cost | Helped |
Surprising findings
- Greedy mean-based planning (NeMean) performs worst, even failing on Pendulum-v1 and MountainCar.
Limitations
- Theory requires strong assumptions (continuity, bounded energy/Lyapunov drift, RKHS dynamics) and the practical algorithm deviates from the theoretical one (fixed horizon, MPC, optimization over M_n).
- Experiments use simulators with limited environment diversity; no real-world experiments reported.
Experimental setup
Datasets: OpenAI gym Pendulum-v1 MountainCar DeepMind control Cartpole DeepMind control Reacher DeepMind control Swimmer racecar simulator soft robotic arm simulator
Benchmarks: Pendulum-v1 MountainCar Cartpole Reacher Swimmer RaceCar SoftArm CartPoleBalance (with resets)
Metrics: normalized average cost cumulative regret number of resets
Baselines: NeMean NePETS NeTS
Builds on: H-UCRL PETS UCRL2 iCEM
Closest papers by meaning
- SOMBRL: Scalable and Optimistic Model-Based RL (NeurIPS 2025 · similarity 0.92)
- Sample-efficient and Scalable Exploration in Continuous-Time RL (ICLR 2026 · similarity 0.92)
- Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds (ICLR 2025 · similarity 0.90)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.89)
- Minimax Optimal Reinforcement Learning with Quasi-Optimism (ICLR 2025 · similarity 0.89)
- MetaCURL: Non-stationary Concave Utility Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.