Research map / Reinforcement learning / Policy gradient and actor-critic

TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Diagnoses critic-side gradient ill-conditioning in multi-task PPO and proposes TOPPO, combining PopArt, critic LayerNorm, FairGrad (alpha=1) critic aggregation and PCGrad actor aggregation, reaching strong Meta-World MT50 results with ~717K parameters.

Problem

On-policy PPO is underexplored for multi-task RL; vanilla MT-PPO underperforms SAC-family methods on Meta-World, with tail tasks stalling due to critic-side gradient imbalance across tasks.

Approach

Reformulate MT-PPO by targeting the critic-side optimization bottleneck with four stage-specific modules: PopArt value-target normalization, critic-only LayerNorm, FairGrad (alpha=1) critic gradient aggregation with a projected Newton solver, and PCGrad on actor gradients, instead of relying on modular architectures or large models.

Key results

• TOPPO reaches 90.88 ± 1.59% final mean success on Meta-World V2 MT50 with 717K parameters, 1.98 points above ARS-LN-1024 with 22.7x fewer parameters.
• TOPPO reaches 56.50 ± 7.76% on the worst-10 MT50 tail tasks, 27.2 points above ARS-LoRA with 22.5x fewer parameters.
• On Meta-World V1 MT50, TOPPO reaches 79.3% IQM, 17.6 points above the strongest published V1 baseline MOORE (61.8%) at 14.5x fewer parameters.

Surprising findings

• Vanilla MT-PPO with a [400x3] backbone reaches 85.2% on MT10 V2, far above the ~30% originally reported for MT-PPO, suggesting earlier conclusions underestimated PPO.
• Critic per-task gradient-norm spread (~497x) is far larger than actor spread (~4.1x) after per-task advantage normalization.

Limitations

• FG-c is not shown to empirically dominate other critic-side combiners; FG-c and PCGrad-c tie within seed noise.
• Actor-side aggregator choices were not exhaustively tested; PCGrad-a is a working choice.
• PopArt uses Welford full-history statistics rather than the canonical EMA, which may be suboptimal under non-stationary returns.

Experimental setup

Datasets: Meta-World V2 Meta-World V1

Benchmarks: Meta-World V2 MT10 Meta-World V2 MT50 Meta-World V1 MT50

Metrics: success rate worst-k tail-task success rate IQM success rate

Baselines: SAC-MT MT-MH-SAC Soft Modularization PCGrad-SAC PaCo MOORE SMT ARS ARS-LN ARS-LoRA Vanilla MT-PPO

Builds on: PPO PopArt FairGrad PCGrad Meta-World+

Closest papers by meaning

• Towards Convergence of PPO: An Approximate Descent Approach (NeurIPS 2026 · similarity 0.90)
• PSPO: Prompt-Level Prioritization and Experience-Weighted Smoothing for Efficient Policy Optimization (AAAI 2026 · similarity 0.89)
• ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages (ICML 2024 · similarity 0.88)
• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.87)
• TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.87)
• Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.87)