Research map / Reinforcement learning / Policy gradient and actor-critic
Identifying Policy Gradient Subspaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically shows that policy gradients of PPO and SAC actor and critic lie in a low-dimensional, slowly changing high-curvature Hessian subspace across 12 continuous-control benchmarks.
Problem
Whether the gradient subspace structure found in supervised learning (gradients lie in a low-dimensional, slowly changing high-curvature subspace) also holds for deep policy gradient RL despite non-stationary data distributions.
Approach
Transfers the supervised-learning gradient-subspace analysis to deep policy gradient methods, measuring gradient subspace fraction and subspace overlap using top-100 Hessian eigenvectors (via Lanczos) for PPO and SAC actors and critics, including mini-batch approximations and suboptimal hyperparameters.
Key results
- For PPO, the gradient subspace fraction in the top-100 Hessian subspace (about 2% of parameters) is well above the random baseline of 0.02, with the critic subspace fraction consistently higher than the actor's.
- Subspace overlap between t1=100k and later timesteps stays considerably above zero even after 400k timesteps for both PPO and SAC.
- SAC shows a small overlap drop at about 1.1M steps when the replay buffer is fully replaced, indicating data distribution affects the subspace.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the top-100 Hessian eigenvectors as the subspace | PPO and SAC actor and critic gradients · Ant, Finger-spin, LunarLanderContinuous, Walker2D | gradient subspace fraction | Helped |
| Use the top-100 Hessian eigenvectors as the subspace | PPO and SAC critics · Ant, Finger-spin, LunarLanderContinuous, Walker2D | gradient subspace fraction | Helped |
| Use SAC instead of PPO for the actor | Actor gradient subspace · Ant, Finger-spin, LunarLanderContinuous, Walker2D | gradient subspace fraction | Helped |
| Estimate gradients with mini-batches instead of large datasets | PPO and SAC actor and critic subspace analysis · Ant, Finger-spin, LunarLanderContinuous, Walker2D | gradient subspace fraction | Hurt |
| Estimate the subspace with a mini-batch Hessian instead of a precise Hessian | PPO and SAC actor and critic · Ant, Finger-spin, LunarLanderContinuous, Walker2D | gradient subspace fraction | Helped |
| Reuse an early high-curvature subspace during training | PPO and SAC actor and critic · 12 benchmark tasks | subspace overlap | Helped |
| Use a longer interval between subspace identification and evaluation | PPO and SAC actor and critic · 12 benchmark tasks | subspace overlap | Hurt |
| Replace the original replay-buffer data with newly collected data | SAC subspace · Ant, HalfCheetah, Swimmer, Walker2D | subspace overlap | Hurt |
| Sample random hyperparameter configurations instead of using tuned hyperparameters | PPO actor · Finger-spin and Walker2D | gradient subspace fraction | Mixed |
| Sample random hyperparameter configurations instead of using tuned hyperparameters | SAC actor · Finger-spin and Walker2D | gradient subspace fraction | Helped |
| Sample random hyperparameter configurations instead of using tuned hyperparameters | PPO and SAC critics · Finger-spin and Walker2D | gradient subspace fraction | No clear effect |
Surprising findings
- For SAC's actor, the gradient subspace fraction is highest in early training and trends in reverse compared to PPO.
- Random hyperparameter configurations yield a large spread in PPO actor gradient subspace fraction, while SAC's is more stable.
Limitations
- Analysis limited to two algorithms (PPO, SAC) and simulated benchmark tasks.
- Hessian estimated from large data sets rather than true Hessian; mini-batch gradient falls partly outside subspace.
- PPO actor subspace fraction sensitive to hyperparameters.
Experimental setup
Datasets: OpenAI Gym Gym Robotics DeepMind Control Suite
Benchmarks: Ant Ball_in_cup BipedalWalker FetchReach Finger-spin HalfCheetah Hopper LunarLanderContinuous Pendulum Reacher Swimmer Walker2D
Metrics: gradient subspace fraction subspace overlap Hessian eigenvalue spectrum maximum episode return
Builds on: PPO SAC Stable Baselines3 RL Baselines3 Zoo
Closest papers by meaning
- Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator (ICML 2018 · similarity 0.89)
- Trajectory-Based Off-Policy Deep Reinforcement Learning (ICML 2019 · similarity 0.88)
- Reusing Trajectories in Policy Gradients Enables Fast Convergence (ICML 2026 · similarity 0.88)
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.88)
- Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States (ICML 2024 · similarity 0.87)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.