Research map / Reinforcement learning / Visual and representation RL

The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: Learned state and action representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirically analyzes policy parameter evolution in TD3/RAD via temporal SVD and proposes PPTB, a plug-in that trims minor SVD directions and boosts major ones of the policy path, improving TD3, RAD, and DoubleDQN.

Problem

Understanding how policy network parameters evolve during deep RL training, and whether that structure (low-dimensional, monotone major directions, oscillating minor directions) can be exploited to improve sample efficiency and stability of continuous-control and discrete-action agents.

Approach

Stack historical policy parameters into a temporal matrix and apply SVD; keep only the top-r_t major singular directions (trimming minor, detour-prone directions) and extrapolate the first two major coordinates (boosting), applied to the current policy as a plug-in with about 10 lines of code.

Key results

• PPTB on TD3 improves MuJoCo aggregate score to 1.1436 and AUC to 1.2419 relative to TD3 (e.g., Walker2d score 3406 to 4385).
• PPTB on RAD improves DMC aggregate AUC to 1.5480 at 100K and 1.1086 at 500K steps.
• PPTB on DoubleDQN raises MinAtar SpaceInvader score from 77.63 to 99.08.

Measured effects reported in the paper

Surprising findings

• Trimming to small rank (<=64) often improves performance after reconstruction, even without retraining.
• Policy parameter updates are highly asymmetric: most parameters in the first layer change far more than others, and minor temporal SVD directions show harmonic-like detours.

Limitations

• No theoretical justification for the observed phenomena.
• Only TD3 and RAD (with SAC inside) and DoubleDQN are studied; on-policy methods like PPO are not covered.
• Fixed dimensionalities and intervals are used; no adaptive selection, and sparse-reward environments are not evaluated.

Experimental setup

Datasets: MuJoCo DeepMind Control Suite MinAtar D4RL halfcheetah-medium-replay

Benchmarks: MuJoCo DeepMind Control Suite MinAtar

Metrics: score AUC normalized aggregate score

Baselines: TD3 RAD DoubleDQN

Builds on: TD3 RAD DoubleDQN Shrink-and-Perturb

Closest papers by meaning

• Analyzing Generalization in Policy Networks: A Case Study with the Double-Integrator System (AAAI 2024 · similarity 0.89)
• Learning Dynamics and Generalization in Deep Reinforcement Learning (ICML 2022 · similarity 0.89)
• CEM-RL: Combining evolutionary and gradient-based methods for policy search (ICLR 2019 · similarity 0.89)
• ERL-Re$^2$: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation (ICLR 2023 · similarity 0.88)
• Efficient Reinforcement Learning by Discovering Neural Pathways (NeurIPS 2024 · similarity 0.88)
• Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn (NeurIPS 2024 · similarity 0.88)