Research map / Reinforcement learning / Policy gradient and actor-critic

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies representation collapse (feature rank decline, dead neurons, capacity loss) in PPO on Atari and MuJoCo, links it to breakdown of PPO's clipped trust region and performance collapse, and proposes Proximal Feature Optimization (PFO), an auxiliary loss regularizing pre-activation changes.

Problem

PPO agents suffer performance collapse under non-stationarity, and it is unclear whether representation degradation (feature rank collapse, capacity loss) affects on-policy policy optimization and why PPO's trust region fails to prevent it.

Approach

Extends the representation-plasticity analysis from off-policy value-based RL to PPO actors, shows that poor representations make PPO's clipping ineffective (updates on other states move probabilities outside the trust region), and proposes PFO, an L2 penalty on the change in penultimate pre-activations between the current policy and the policy that collected the batch.

Key results

• PFO reduces dead neurons, pre-activation norm and capacity loss and increases feature rank, and removes performance collapse on ALE/NameThisGame and ALE/Phoenix.
• PPO policy feature rank declines in five of six ALE games and seven of eight MuJoCo tasks, with pre-activation norms consistently growing.
• Sharing the actor-critic trunk helps on dense-reward ALE games but collapses performance on sparse-reward Gravitar.

Measured effects reported in the paper

Surprising findings

• The performance collapse yields a high-entropy policy that is trivially constant across states, since all feature neurons die and the output depends only on the bias.
• Feature norm can stay stable while pre-activation norm blows up as neurons die.

Limitations

• Results limited to small PPO architectures without normalization layers, weight decay, or recurrence.
• PFO coefficient not tuned; chosen only by matching loss magnitude.
• Causal connection between representation, trust region and collapse holds mainly around the collapse regime.

Experimental setup

Datasets: Arcade Learning Environment (Atari-5 subset plus Gravitar) MuJoCo (Ant, HalfCheetah, Humanoid, Hopper)

Benchmarks: ALE/Phoenix-v5 ALE/NameThisGame-v5 ALE/Gravitar-v5 MuJoCo Ant, HalfCheetah, Humanoid, Hopper

Metrics: episodic return approximate feature rank capacity loss dead neurons pre-activation norm excess probability ratio

Baselines: PPO-Clip baseline (no intervention) shared actor-critic trunk Adam moment resetting Adam beta2 decay adaptation

Builds on: PPO GAE feature rank collapse analysis in value-based RL capacity loss (target-fitting) metric

Closest papers by meaning

• The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning (ICLR 2026 · similarity 0.87)
• Implementation Matters in Deep RL: A Case Study on PPO and TRPO (ICLR 2020 · similarity 0.87)
• A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
• Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation (ICML 2024 · similarity 0.85)
• Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning (ICLR 2025 · similarity 0.85)
• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.85)