Research map / Reinforcement learning / Policy gradient and actor-critic
Generalizable Policy Improvement via Reinforcement Sampling (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reinforcement Sampling (RS), which improves the action distribution on the visited states using a generalizable action-value function (via NPG or sign policy gradient) before sampling, and distills it into the policy, outperforming fine-tuned PPO on MinAtar.
Problem
Policy gradient methods refine policies only on sampled states and generalize poorly to unseen states; the paper seeks generalizable policy improvement in on-policy RL.
Approach
Instead of improving the policy parameters on historical states, RS improves the action distribution on the currently visited state using a closed-form NPG or sign-policy-gradient update driven by a learned advantage, samples from the improved distribution, and then distills it into the policy via KL minimization; a feature-norm regularizer is added to value learning for better generalization.
Key results
- SignPG-RS outperforms fine-tuned PPO on all five MinAtar games on both overall and last scores (e.g., Asterix last 98.1 vs 22.1; Space Invaders overall 264.2 vs 63.1).
- Results averaged over 5 random seeds with 10M training samples per environment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Asterix (overall) | average undiscounted return: 7.4 ± 0.2 -> 14.5 ± 0.9 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Asterix (last) | average undiscounted return: 22.1 ± 2.2 -> 98.1 ± 8.7 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Breakout (overall) | average undiscounted return: 12.1 ± 0.2 -> 28.6 ± 1.7 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Breakout (last) | average undiscounted return: 679.3 ± 15.0 -> 2287.2 ± 473.5 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Freeway (overall) | average undiscounted return: 29.6 ± 1.3 -> 51.3 ± 1.2 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Freeway (last) | average undiscounted return: 48.9 ± 1.2 -> 62.2 ± 0.2 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Seaquest (overall) | average undiscounted return: 7.3 ± 1.2 -> 13.6 ± 1.5 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Seaquest (last) | average undiscounted return: 125.9 ± 28.5 -> 372.2 ± 46.9 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Space Invaders (overall) | average undiscounted return: 63.1 ± 2.2 -> 264.2 ± 39.7 | Helped |
| replace fine-tuned PPO with SignPG-RS | policy learning on MinAtar · MinAtar Space Invaders (last) | average undiscounted return: 1053 ± 18.0 -> 4601.2 ± 1359.0 | Helped |
Limitations
- Evaluated only on MinAtar, a small set of five games.
- Relies on the action-value function being accurate; RS gains depend on value estimation quality.
- NPG-RS results deferred to an appendix and not in main comparison.
Experimental setup
Datasets: MinAtar Asterix MinAtar Breakout MinAtar Freeway MinAtar Seaquest MinAtar Space Invaders
Benchmarks: MinAtar
Metrics: overall average undiscounted return last-100-epoch average undiscounted return
Baselines: PPO
Builds on: Natural Policy Gradient PPO performance difference lemma feature regularization for deep RL generalization (Lan et al. 2022)
Closest papers by meaning
- Low-Switching Policy Gradient with Exploration via Online Sensitivity Sampling (ICML 2023 · similarity 0.90)
- Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.88)
- On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.88)
- Constructing an Optimal Behavior Basis for the Option Keyboard (NeurIPS 2025 · similarity 0.88)
- How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.