Research map / Reinforcement learning / MDPs and planning
Exploiting the Replay Memory Before Exploring the Environment: Enhancing Reinforcement Learning Through Empirical MDP Iteration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Empirical MDP Iteration (EMIT), which alternates solving an in-sample (empirical MDP) IQL-learned Q-bar function with growing the replay data, and uses it as a regularizer and exploration bonus for DQN and TD3.
Problem
Bellman optimality updates over the full MDP suffer estimation errors when the replay memory covers only part of the state-action space, especially with function approximation.
Approach
Learn an in-sample Q-bar (via IQL-style in-sample Bellman updates) on the empirical MDP defined by replay data, regularize the standard Q-learning loss toward Q-bar, and use |Q - Q-bar| as an exploration bonus so the data grows iteratively toward the original MDP.
Key results
- EMIT on Atari (20 games) and MuJoCo (8 tasks) with normalized scores achieves the best average performance among compared methods (Fig. 4).
- EMIT either outperforms or is within 10% of the best baseline on 19 of 28 tasks.
- Regularizing Q toward the empirical Q-bar keeps the value difference Vdiff near zero on Breakout and HalfCheetah, versus clear over/underestimation for DQN and TD3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply EMIT with IQL to DQN | DQN · Atari | mean score | Helped |
| Apply EMIT with IQL to TD3 | TD3 · MuJoCo | mean score | Helped |
| Use EMIT | DQN and TD3 · Atari and MuJoCo tasks | normalized score: 19 out of 28 tasks | Helped |
| Add Q-function loss regularization toward the empirical-MDP value function (ablation) | EMIT · Breakout and HalfCheetah | mean score | Helped |
| Add the value-difference exploration bonus (ablation) | EMIT · Breakout and HalfCheetah | mean score | Helped |
| Add both loss regularization and the value-difference exploration bonus (ablation) | DQN and TD3 · Breakout and HalfCheetah | mean score | Helped |
Surprising findings
- A passively learned in-sample IQL policy trained on data from an active DQN/TD3 agent matches or exceeds that agent's performance.
- Removing the regularization term hurts more than removing the exploration bonus.
Limitations
- Learns two Q functions, roughly doubling wall-clock time.
- Q-bar policy lacks its own exploration mechanism, so exploration relies on the Q-difference bonus.
- Theoretical guarantees are proven only for the tabular case.
Experimental setup
Datasets: Atari (20 games, ALE) MuJoCo (8 tasks) CliffWalk (toy)
Benchmarks: Atari MuJoCo CliffWalk
Metrics: mean score normalized score value difference Vdiff policy change
Baselines: DQN C51 IQN Rainbow SAC TD3 XQL TRPO PPO
Builds on: DQN TD3 Implicit Q-learning Double Q-learning Bellman optimality equation
Closest papers by meaning
- Replay Memory as An Empirical MDP: Combining Conservative Estimation with Experience Replay (ICLR 2023 · similarity 0.90)
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge (AAAI 2024 · similarity 0.87)
- A Computationally Efficient Algorithm for Infinite-Horizon Average-Reward Linear MDPs (ICML 2025 · similarity 0.87)
- Randomized Exploration for Reinforcement Learning with Multinomial Logistic Function Approximation (NeurIPS 2024 · similarity 0.87)
- Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.