Research map / Reinforcement learning / Policy gradient and actor-critic

Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically (infinite-width linearised two-layer policy with semi-gradient actor-critic) that attainable states lie near a manifold whose dimension is bounded by about 2·d_a+1, and uses a sparsifying local manifold layer in SAC to improve high-DoF MuJoCo/DM Control performance.

Problem

Most RL theory assumes finite state and action spaces; it is unclear how the geometry of continuous state spaces relates to the dimensionality of the action space under neural policy training.

Approach

Treat the set of states locally attained by a linearised wide two-layer policy trained by semi-gradient updates as a low-dimensional manifold via Lie series expansion and infinite-width limit analysis, bounding its dimension linearly in action dimension; then replace one feed-forward layer with a CRATE-style sparsification layer in SAC.

Key results

• Estimated intrinsic dimension of attainable states stays below 2d_a+1 for four MuJoCo environments.
• In a toy linear system with d_a=1, the attained-set dimension remains at or below 2d_a+1=3 for embedding dimensions 3 to 10.
• SAC with one sparsification layer (width 1024) outperforms vanilla SAC on high-dimensional Dog environments, where vanilla SAC fails to learn.

Measured effects reported in the paper

Surprising findings

• Vanilla SAC fails to learn on high-dimensional Dog tasks while the sparsified variant learns well.

Limitations

• Theory assumes deterministic transitions and infinite-width, linearised policies with access to the true Q function.
• Analysis does not extend to very high-dimensional state spaces where d_s grows with n.

Experimental setup

Datasets: MuJoCo environments (Cheetah, and three others) DM Control Ant, Dog Stand, Dog Walk, Quadruped Walk toy linear control system

Benchmarks: MuJoCo DM Control Ant DM Control Dog Stand DM Control Dog Walk DM Control Quadruped Walk toy linear control environment

Metrics: intrinsic dimensionality estimate discounted return return difference between linearised and canonical policies

Baselines: DDPG SAC canonical two-layer NN policy

Builds on: DDPG SAC CRATE Lie series / control-affine systems wide neural network NTK-style linearisation

Closest papers by meaning

• From Parameters to Behaviors: Unsupervised Compression of the Policy Space (ICLR 2026 · similarity 0.89)
• Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control (ICML 2018 · similarity 0.88)
• From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments (ICLR 2026 · similarity 0.88)
• Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learning (ICLR 2026 · similarity 0.88)
• Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces (NeurIPS 2024 · similarity 0.86)
• Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking (NeurIPS 2024 · similarity 0.86)