Research map / Reinforcement learning / Visual and representation RL
Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether decoupling actor and critic representations in on-policy deep RL (PPO, PPG, DCPG) causes them to specialise in different information, and how auxiliary representation objectives shape each.
Problem
Actor-critic agents learning from high-dimensional observations often share one representation, but it is unclear what information the actor and critic should each extract; prior work reported gains from decoupling without a clear explanation.
Approach
Proposes mutual-information metrics (overfitting I(Z;L), value I(Z;V), latent dynamics I(Z;Z'), and I(Z,Z';A)) and a theoretical account of optimal actor/critic representations, showing that decoupling induces information specialisation: the actor encodes action-relevant information and the critic encodes value and dynamics information.
Key results
- Decoupling PPO yields 37% higher I(Z_A;V) than shared PPO without value distillation, and decoupled models are more parameter efficient, outperforming a shared model with four times its parameter count.
- Dynamics prediction on the actor increases I((Z,Z');A) and generally improves PPG and DCPG performance, while MICo on the actor tends to increase overfitting-related information I(Z;L).
- Applying MICo to the critic induces a data collection bias that often worsens performance, while combined actor/critic objectives can produce sharp gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Decouple actor and critic representations | PPO · Procgen | C(Z_A | O;L): 65.2 $\pm$ 3 -> 52.3 $\pm$ 3 | Helped |
| Decouple actor and critic representations | PPG · Procgen | C(Z_A | O;L): 70.0 $\pm$ 2 -> 75.5 $\pm$ 2 | Mixed |
| Decouple actor and critic representations | DCPG · Procgen | C(Z_A | O;L): 77.6 $\pm$ 2 -> 76.4 $\pm$ 2 | No clear effect |
| Add data augmentation to the actor (ablation) | PPO, PPG and DCPG · Procgen | train and test scores | Helped |
| Add dynamics prediction to the actor (ablation) | PPG and DCPG · Procgen | performance | Helped |
| Add dynamics prediction to the actor (ablation) | PPO · Procgen | performance | No clear effect |
| Add MICo to the actor (ablation) | PPO, PPG and DCPG · Procgen | performance | Hurt |
| Add advantage prediction to the actor (ablation) | PPO, PPG and DCPG · Procgen | performance | No clear effect |
| Apply value distillation over large, diverse training batches (ablation) | PPG and DCPG · Procgen | performance | Helped |
| Apply MICo to the critic (ablation) | PPO · Procgen | performance | Hurt |
| Combine advantage prediction on the actor with MICo on the critic (ablation) | PPO · Procgen | performance | Helped |
| Decouple actor and critic representations | PPO · Procgen | I(Z_A;V): +37% | Helped |
Surprising findings
- Decoupled PPO is more parameter efficient than shared PPO, beating a shared model with four times its parameters.
- Neither actor nor critic representations benefit from being Markov; no significant correlation was found between the sum of the Markov-related mutual information terms and agent performance.
Limitations
- Study limited to on-policy algorithms (PPO, PPG, DCPG) and specific benchmarks (Procgen, Brax)
- Mutual information estimated with k-NN estimators on subsampled batches, and the estimates can underestimate I(O;.) for high-dimensional observations
- Results for PPG and DCPG cannot formally confirm implicit knowledge transfer because explicit value distillation also occurs
Experimental setup
Datasets: Procgen Brax
Benchmarks: Procgen Brax video distractors
Metrics: normalised train score normalised test score mutual information I(Z;.) compression efficiency parameter scaling performance
Baselines: shared-representation PPO shared-representation PPG shared-representation DCPG MICo dynamics prediction data augmentation advantage prediction
Builds on: PPO PPG DCPG MICo Procgen
Closest papers by meaning
- Representation-Driven Reinforcement Learning (ICML 2023 · similarity 0.89)
- Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Critic (NeurIPS 2021 · similarity 0.88)
- Learning Actionable Representations with Goal Conditioned Policies (ICLR 2019 · similarity 0.88)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.88)
- Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.86)
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.