Research map / Reinforcement learning / Policy gradient and actor-critic
Occupancy-based Policy Gradient: Estimation, Convergence, and Optimality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops model-free occupancy-based policy gradient methods (OCCUPG online, OFF-OCCUPG offline with smooth clipping) that estimate gradients via squared-loss regression, with local and global convergence and sample complexity guarantees, and extend to general differentiable functionals.
Problem
Policy optimization using only occupancy functions (not value functions) is computationally efficient and provably convergent, both online (exploration via initial distribution) and offline (poor data coverage), and for general objectives beyond expected return.
Approach
Recursive Bellman-flow-like expression for the gradient of log occupancies reduces occupancy-gradient estimation to a sequence of squared-loss regressions; offline, a smooth-clipping function replaces hard clipping so plug-in occupancy estimates yield statistically robust gradients.
Key results
- OCCUPG reaches an epsilon-stationary point with T=O(beta H/epsilon) iterations and n=O~(p d_G H^6 G^2 log(T/delta)/epsilon) samples per iteration, using O(beta H^2/epsilon) regression oracle calls.
- OFF-OCCUPG estimates the offline smooth-clipped gradient with sample complexity scaling with (sum_h C^s_h C^a_h)^2 L_sigma^2 and converges to a stationary point with T=O(beta H^2/epsilon^2).
- Proposition 4.2 shows hard clipping of plug-in occupancies yields O(1) gradient error even when occupancy estimates are arbitrarily accurate, motivating smooth clipping.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Estimate occupancy gradients recursively with squared-loss regression | OCCUPG · Online finite-horizon MDP | gradient estimation sample complexity: n = eO\left(\frac{pd_GH^6G^2\log(1/\delta)}{\varepsilon^2}\right) | Helped |
| Optimize with exploratory initial distribution ν under the stated policy completeness and gradient domination conditions | OCCUPG · Online finite-horizon MDP | iterations for ε-optimality: T = eO\left(\frac{\beta B^2(C_{\pi^*})^2m^2H^2}{\varepsilon^2}\right) | Helped |
| Replace hard clipping with smooth clipping | OFF-OCCUPG · Offline finite-horizon MDP | plug-in occupancy-gradient estimation robustness | Helped |
| Use hard clipping instead of smooth clipping (ablation) | OFF-OCCUPG · Offline finite-horizon MDP | plug-in occupancy-gradient error: O(1) | Hurt |
| Replace hard clipping with smooth clipping | OFF-OCCUPG · Offline finite-horizon MDP | objective bias: ≲ H^2D_σ | Mixed |
| Use a plug-in occupancy estimate to evaluate the objective derivative | OCCUPG for general functionals · General differentiable occupancy functionals | gradient estimation error: \|\nabla J_F(\pi)-\widehat{\nabla J_F(\pi)}\|\lesssim H^2GL_F\sqrt{\frac{p\log(2pH|F|/\delta)}{n}}+C_F\sqrt{\frac{pd_GH^ | Helped |
Surprising findings
- Hard clipping (used in prior occupancy work) can produce constant-order gradient error even with arbitrarily accurate occupancy estimates.
Limitations
- Purely theoretical; no empirical experiments.
- Offline global optimality only holds under strong all-policy data coverage.
- Offline gradient function class assumption (Asm. 4.2) is stronger than the online one.
Experimental setup
Metrics: sample complexity gradient mapping norm suboptimality
Builds on: FORC Bellman completeness / FQI PG theorem with gradient domination (Agarwal et al. 2021) clipped occupancy of Huang et al. 2023 Liu et al. 2019 off-policy PG
Closest papers by meaning
- On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Global Optimality for Constrained Exploration via Penalty Regularization (NeurIPS 2026 · similarity 0.90)
- A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- OMPO: A Unified Framework for RL under Policy and Dynamics Shifts (ICML 2024 · similarity 0.88)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.