Research map / Reinforcement learning / Policy gradient and actor-critic
Analyzing Generalization in Policy Networks: A Case Study with the Double-Integrator System
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper analyzes why DRL policy networks degrade when the state norm grows, using a state-division analysis on the double-integrator system, and shows that tanh saturation turns the division boundary linear, causing bang-bang-like control with unavoidable overshoot.
Problem
Performance degradation of DRL policy networks in expanded state spaces (larger state norm than training), where the underlying cause is poorly understood.
Approach
Uses state division as an intrinsic analysis tool rather than a post-hoc explanation: as the state norm grows, tanh saturates, so the nonlinear division boundary of the first layer becomes linear, producing bang-bang-like control that inevitably overshoots on the double-integrator task.
Key results
- Across DDPG, TD3, SAC and PPO agents, the division boundary of the best-performing policies becomes essentially linear in the far state space, indicating the phenomenon is independent of the optimization algorithm.
- A flatter division line causes agents starting far from the origin (e.g., at -40) to decelerate late, producing unavoidable overshoot relative to ideal bang-bang control.
- Failed policies were found to contain dead zones created by radial division lines in which trajectories diverge from the origin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a saturating tanh policy network in an expanded state space | DRL policies for the double-integrator system · Expanded state space | overshooting | Hurt |
| Use different DRL optimization algorithms | Three-layer, 32-node policy networks · Double-integrator control with initial positions -10, -20 and -40 | state-division linearity | No clear effect |
| Use a steeper division line | Policy network for double-integrator control · Initial positions -20 and -40 | overshooting | Hurt |
| Use a flatter division line | Policy network for double-integrator control · Initial positions -10 and -20 | convergence to the origin | Mixed |
| Enter a dead zone formed by radial division lines | Policy network for double-integrator control · Failed policy network | target convergence | Hurt |
| Shift the state across a significant division strip | Policy network with division line significance 9.15 · Division strip example (Fig. 8) | feature offset: 0.55 | Helped |
Surprising findings
- Overshoot under large initial error is attributed to network saturability itself rather than the optimization objective, since DRL optimizes discounted reward rather than time.
Limitations
- Analysis is restricted to the one-dimensional double-integrator system and a small three-layer, 32-node tanh network.
- Extension to high-dimensional state spaces and hyperplane division is left open.
- Linear division for general ReLU networks is observed but its predictability is not established.
Experimental setup
Datasets: double-integrator control environment
Benchmarks: double-integrator control
Metrics: position response overshoot state-action pattern
Baselines: DDPG TD3 SAC PPO
Builds on: DDPG TD3 SAC PPO
Closest papers by meaning
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.89)
- Regularization Matters in Policy Optimization - An Empirical Study on Continuous Control (ICLR 2021 · similarity 0.89)
- Discovering symbolic policies with deep reinforcement learning (ICML 2021 · similarity 0.87)
- Towards Generalization and Simplicity in Continuous Control (NeurIPS 2017 · similarity 0.87)
- What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (ICLR 2021 · similarity 0.87)
- Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.