Research map / Reinforcement learning / Policy gradient and actor-critic
Prioritized Generative Replay
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prioritized Generative Replay (PGR), which trains a conditional diffusion model on the online replay buffer and guides generation with relevance functions (curiosity, return, TD-error) to densify and prioritize synthetic transitions for off-policy RL.
Problem
Online RL replay buffers replay uniformly, yet relevant transitions are rare; prioritized replay can overfit, and unconditional generative replay does not target the most useful transitions.
Approach
Model the replay buffer as a parametric conditional diffusion model whose conditioning signal is a relevance function (chiefly an ICM-style curiosity prediction error), so generated transitions densify and focus on learning-relevant, novel regions rather than merely reweighting real data.
Key results
- On state-based DMC-100K, PGR (Curiosity) reaches 927.98 on Quadruped-Walk versus 727.01 for SynthER and 496.75 for REDQ.
- On Pixel-DMC-100K Cheetah-Run, PGR (Curiosity) scores 529.70 versus 489.30 for DrQ-v2 and 465.09 for SynthER.
- On OpenAI Gym, PGR (Curiosity) achieves 5682.33 on Walker2d-v2 versus 4829.32 for SynthER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use curiosity-guided generative replay instead of unconditional replay | SynthER on DMC-100k Quadruped-Walk · DMC-100k Quadruped-Walk | average return: 727.01 $\pm$ 86.66 -> 927.98 $\pm$ 25.18 | Helped |
| use curiosity-guided generative replay instead of unconditional replay | SynthER on DMC-100k Cheetah-Run · DMC-100k Cheetah-Run | average return: 729.35 $\pm$ 49.59 -> 817.36 $\pm$ 35.93 | Helped |
| use curiosity-guided generative replay instead of unconditional replay | SynthER on DMC-100k Reacher-Hard · DMC-100k Reacher-Hard | average return: 838.60 $\pm$ 131.15 -> 915.21 $\pm$ 48.24 | Helped |
| use curiosity-guided generative replay instead of REDQ with curiosity bonus | REDQ on DMC-100k Finger-Turn-Hard · DMC-100k Finger-Turn-Hard | average return: 777.66 $\pm$ 116.96 -> 885.98 $\pm$ 67.29 | Helped |
| use curiosity-guided generative replay instead of DrQ-v2 | DrQ-v2 on Pixel-DMC-100k Walker-Walk · Pixel-DMC-100k Walker-Walk | average return: 514.11 $\pm$ 81.42 -> 570.99 $\pm$ 41.44 | Helped |
| use curiosity-guided generative replay instead of DrQ-v2 | DrQ-v2 on Pixel-DMC-100k Cheetah-Run · Pixel-DMC-100k Cheetah-Run | average return: 489.30 $\pm$ 69.26 -> 529.70 $\pm$ 27.76 | Helped |
| use curiosity-guided generative replay instead of the strongest listed baseline | model-free/model-based baselines on OpenAI Gym tasks · OpenAI Gym Walker2d-v2, HalfCheetah-v2, and Hopper-v2 | average return: 4829.32 $\pm$ 191.16 -> 5682.33 $\pm$ 370.04; 8612.49 $\pm$ 407.53 -> 9234.61 $\pm$ 658.77; 3395.21 $\pm$ 117.50 -> 4101 | Helped |
| condition generation on reward instead of curiosity (ablation) | PGR on DMC-100k Quadruped-Walk · DMC-100k Quadruped-Walk | average return: 927.98 $\pm$ 25.18 -> 510.39 $\pm$ 121.11 | Hurt |
| condition generation on return instead of curiosity (ablation) | PGR on DMC-100k Quadruped-Walk · DMC-100k Quadruped-Walk | average return: 927.98 $\pm$ 25.18 -> 737.62 $\pm$ 20.13 | Hurt |
| condition generation on TD error instead of curiosity (ablation) | PGR on DMC-100k Quadruped-Walk · DMC-100k Quadruped-Walk | average return: 927.98 $\pm$ 25.18 -> 802.18 $\pm$ 116.52 | Hurt |
| use RND relevance instead of intrinsic curiosity (ablation) | PGR on Pixel-DMC Walker-Walk · Pixel-DMC-100k Walker-Walk | average return: 570.99 $\pm$ 41.44 vs 602.10 $\pm$ 43.44 | Mixed |
| use CTS pseudo-count relevance instead of intrinsic curiosity (ablation) | PGR on Pixel-DMC Cheetah-Run · Pixel-DMC-100k Cheetah-Run | average return: 529.70 $\pm$ 27.76 -> 508.17 $\pm$ 74.03 | Hurt |
Surprising findings
- Naively conditioning on reward (Reward-PGR) performs worse than unconditional SynthER on several DMC tasks.
- Both PGR and SynthER fail catastrophically when the synthetic data ratio is raised to 0.875.
Limitations
- Fails at very high synthetic data ratios (0.875), so real environment interaction is still needed.
- Evaluated mainly on DMC-100K and OpenAI Gym, which the authors note are not well suited to studying exploration.
- Curiosity-based gains in stochastic settings were tested only on DMLab with PPO rather than the main off-policy backbone.
Experimental setup
Datasets: DeepMind Control Suite online replay (100K steps) OpenAI Gym v2 online replay DMLab randomized mazes
Benchmarks: DMC-100K state-based Pixel-DMC-100K OpenAI Gym Walker2d-v2, HalfCheetah-v2, Hopper-v2 DMLab Sparse and Very Sparse noisy-observation DMC
Metrics: average return sample efficiency model MSE for generated transitions dormant ratio model size and runtime
Baselines: SAC REDQ MBPO Dreamer-V3 SynthER DrQ-v2 PER (TD-Error) PER (Curiosity) MAX PPO NoisyNets Boot-DQN
Builds on: SynthER prioritized experience replay intrinsic curiosity module classifier-free guidance REDQ
Closest papers by meaning
- Synthetic Experience Replay (NeurIPS 2023 · similarity 0.90)
- Reliability-Adjusted Prioritized Experience Replay (ICLR 2026 · similarity 0.89)
- Prioritizing Samples in Reinforcement Learning with Reducible Loss (NeurIPS 2023 · similarity 0.89)
- From Generative to Episodic: Sample-Efficient Replicable Reinforcement Learning (ICML 2026 · similarity 0.89)
- Prioritized Level Replay (ICML 2021 · similarity 0.89)
- A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.