Research map / Reinforcement learning / Offline RL

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Behavior regularization for offline RL

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes when off-policy policy gradient deployment for LLM post-training is justified and what roles importance sampling, KL regularization, and baselines play in correcting the resulting distribution mismatch.

Problem

Policy gradient methods used for LLM post-training are routinely run off-policy, although vanilla policy gradient assumes on-policy sampling; it is unclear when this is justified and what the core correction ingredients do.

Approach

Analytical study of off-policy policy gradient that dissects the roles of importance sampling, KL regularization, and baselines in correcting the distribution mismatch between sampling and current policy.

Experimental setup

Builds on: policy gradient importance sampling

Closest papers by meaning

• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.92)
• VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.91)
• Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.91)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.90)
• Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.90)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.90)