Research map / Reinforcement learning / Policy gradient and actor-critic
Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SUFT, a causal upper bound on the on-policy (factual) loss expressed via off-policy loss plus an off-policy-evaluation term computed from stored behavior-policy value outputs in the replay buffer, improving sample efficiency of DQN, Double DQN, PPO and SAC.
Problem
Off-policy DRL agents need large replay buffers and many training steps because of the mismatch between behavior and target policies, hurting sample efficiency and stability.
Approach
Derives a Neyman-Rubin potential-outcomes bound that upper-bounds the unobserved factual (on-policy) loss by the observed counterfactual (off-policy) loss plus an estimated treatment effect term (SUFT OPE), and computes that term cheaply by storing behavior-policy Q/V outputs in the replay buffer instead of discarding them.
Key results
- SUFT achieves a 383.19% mean reward ratio for DQN on Atari, 226.91% for Double DQN, 197.34% for PPO, and 224.53% for SAC on MuJoCo.
- With a 4K replay buffer, SUFT DQN reaches a 63.45% human normalized mean, above DQN with a 100K buffer (29.29%).
- SUFT reduces the replay buffer size by up to 96% and outperforms a 25x larger buffer baseline by 437.05% mean reward ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the SUFT OPE term to DQN | DQN · Atari 57 games | mean reward ratio: 383.19% | Helped |
| Add the SUFT OPE term to Vanilla DQN | Vanilla DQN · Atari 57 games | mean reward ratio: 228.12% | Helped |
| Add the SUFT OPE term to Double DQN | Double DQN · Atari 57 games | mean reward ratio: 226.91% | Helped |
| Add the SUFT OPE term to PPO | PPO · Atari 57 games | mean reward ratio: 197.34% | Helped |
| Add the SUFT OPE term to SAC | SAC · MuJoCo five environments | mean reward ratio: 224.53% | Helped |
| Add the SUFT OPE term to PPO | PPO · MuJoCo five environments | mean reward ratio: 111.21% | Helped |
| Use SUFT with a 4K replay buffer instead of the 100K-buffer baseline | DQN · Atari 57 games | mean reward ratio gain: 437.05% | Helped |
| Add the SUFT OPE term to DQN (ablation) | DQN · Atari games | Human Normalized Mean: 25.04% -> 63.45% | Helped |
| Increase the DQN replay buffer from 4K to 100K (ablation) | DQN · Atari games | Human Normalized Mean: 25.04% -> 29.29% | Helped |
| Use Vanilla DQN rather than standard DQN (ablation) | DQN · Atari games | Human Normalized Mean: 25.04% -> 41.26% | Helped |
| Use Double DQN rather than standard DQN (ablation) | DQN · Atari games | Human Normalized Mean: 25.04% -> 42.31% | Helped |
| Add an inertia regularizer instead of the SUFT causal-bound term (ablation) | Double DQN · Atari 57 games | mean reward ratio: 83.21% vs 226.91% | Hurt |
Surprising findings
- An inertia regularizer (squared Q-current minus Q-target) degrades Double DQN to an 83.21% mean reward ratio, suggesting gains are not just from regularization.
Limitations
- The L2-loss case does not satisfy the inverse triangle inequality assumption used in the proof; the authors rely on empirical validation instead.
- Evaluation uses small replay buffers (4K) and limited training steps, so behavior at larger scale is not established in the main text.
- Theory extends to N control treatments via induction, while the DRL reduction relies on identifying behavior policies with control treatments under strong ignorability assumptions.
Experimental setup
Datasets: Atari 2600 (57 games, Gymnasium v5) MuJoCo (5 environments, Gymnasium v4)
Benchmarks: Atari 2600 (57 games) MuJoCo (5 environments)
Metrics: mean reward ratio median smoothed training reward human normalized mean t-test p-value
Baselines: DQN Vanilla DQN Double DQN PPO SAC inertia regularizer double DQN
Builds on: DQN Double DQN PPO SAC Neyman-Rubin potential outcomes framework
Closest papers by meaning
- The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning (ICML 2025 · similarity 0.90)
- Confounding Robust Deep Reinforcement Learning: A Causal Approach (NeurIPS 2025 · similarity 0.87)
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.87)
- ERCI: An Explainable Experience Replay Approach with Causal Inference for Deep Reinforcement Learning (AAAI 2025 · similarity 0.86)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.