Research map / Reinforcement learning / Policy gradient and actor-critic

When Does Non-Uniform Replay Matter in Reinforcement Learning?

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies when non-uniform replay helps in off-policy RL via replay volume, expected recency, and sampling entropy, and proposes a truncated geometric replay sampler that helps in low-replay-volume regimes.

Problem

It is unclear when and why non-uniform replay sampling improves over uniform replay in modern off-policy RL, especially in parallel simulation and multi-task settings where replay volume is low.

Approach

Decomposes replay per environment step into expected recency, replay volume (UTD x batch size), and sampling entropy; shows recency bias matters most at low replay volume and that high-entropy, full-support recency bias beats entropy-restricted samplers like ERE at matched recency. Proposes a truncated geometric sampler with O(1) inverse-CDF sampling.

Key results

• On FastTD3 HumanoidBench (29 tasks), truncated geometric sampling achieves up to 1.35x faster learning than uniform replay in environment steps and reaches comparable performance in about 80% of training time.
• In the multi-task BRC setting, truncated geometric sampling achieves the best overall performance and improves substantially over uniform replay, with ERE competitive but slightly below.
• At high UTD and standard batch size on SimbaV2 (20 tasks), truncated geometric performs on par with uniform replay, a modest 1.2% improvement.

Surprising findings

• Uniform FIFO with a ~300K buffer outperforms Uniform with a 1M buffer at matched-recency comparisons, attributed to higher expected recency.
• ERE, the lowest-entropy recency-biased sampler, underperforms even uniform replay when expected recency is matched.

Limitations

• Empirical study without a complete theory of replay.
• Evaluated only on continuous-control off-policy RL; discrete-action domains such as Atari not studied.
• Fixed alpha=10 across experiments; adaptive recency schedules not explored.

Experimental setup

Datasets: HumanoidBench Isaac Lab DMC Dogs DMC Humanoids Meta-World MyoSuite

Benchmarks: HumanoidBench HumanoidBench-Hard Isaac Lab DMC Dogs Meta-World MyoSuite

Metrics: normalized return success rate normalized AUC wall-clock time

Baselines: Uniform replay PER ERE Uniform FIFO

Builds on: Prioritized Experience Replay Emphasizing Recent Experience FastTD3 SimbaV2 BRC

Closest papers by meaning

• Revisiting Fundamentals of Experience Replay (ICML 2020 · similarity 0.90)
• Replay Memory as An Empirical MDP: Combining Conservative Estimation with Experience Replay (ICLR 2023 · similarity 0.89)
• Replay across Experiments: A Natural Extension of Off-Policy RL (ICLR 2024 · similarity 0.88)
• Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization (ICML 2024 · similarity 0.87)
• Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (AAAI 2024 · similarity 0.87)
• Hybrid Reinforcement Learning from Offline Observation Alone (ICML 2024 · similarity 0.86)