Research map / Reinforcement learning / MDPs and planning
Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Q-Rex with online target learning and reverse experience replay on ZIBEL/linear MDPs | Q-learning · ZIBEL/linear MDP | sample complexity: vanilla Q-learning can fail to converge; Q-Rex finds the optimal policy with sample complexity \tilde{O}\left(\kappa \ma | Helped |
| Use Q-Rex with asynchronous data | Tabular Q-learning · Tabular MDP | sample complexity: \tilde{O}\left(\tfrac{1}{\mu_{\min}}\max\left(\tfrac{1}{(1-\gamma)^4\min(\epsilon,\epsilon^2)},\tfrac{\tmix}{1-\gamma}\r | Helped |
| Reuse data from the first outer-loop iteration (ablation) | Q-Rex · Tabular MDP with bounded transition support | sample complexity: \tilde{O}\left(\tfrac{1}{\mu_{min}}\max\left(\tmix,\tfrac{1}{\epsilon^2(1-\gamma)^3},\tfrac{\bar{d}}{(1-\gamma)^2},\tfra | Helped |
| Use EpiQ-Rex rather than vanilla Q-learning | Q-learning · Mountain Car | mean episode length: 145 -> 136 | Helped |
| Use EpiQ-Rex rather than OTL+ER+Q-learning | Q-learning · Mountain Car | mean episode length: 143 -> 136 | Helped |
| Replace randomly ordered replay with reverse-order replay (ablation) | OTL+ER+Q-learning · 100-State Random Walk | final solution quality | Helped |
| Add online target learning (ablation) | Q-learning · Baird's Counter-Example | convergence | Helped |
| Use Q-Rex rather than vanilla Q-learning | Q-learning · Linear dynamical system with linear rewards | value estimation error | Helped |
| Use Q-Rex rather than OTL+ER-based Q-learning | Q-learning · Linear dynamical system with linear rewards | value estimation error | Helped |
| Use reverse replay with OTL rather than ER+Q-learning (ablation) | Q-learning · Linear dynamical system with linear rewards | value estimation error | Hurt |
Closest papers by meaning
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.89)
- Regularized Q-Learning (NeurIPS 2024 · similarity 0.89)
- Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.89)
- Exploiting the Replay Memory Before Exploring the Environment: Enhancing Reinforcement Learning Through Empirical MDP Iteration (NeurIPS 2024 · similarity 0.89)
- Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge (AAAI 2024 · similarity 0.89)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.