Research map / Reinforcement learning / Policy gradient and actor-critic
Relative Entropy Pathwise Policy Optimization
Claas A Voelcker, Axel Brunnbauer, Marcel Hussing, Michal Nauman, Pieter Abbeel, Radu Grosu, Eric Eaton, Amir-massoud Farahmand, Igor Gilitschenski
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace pathwise Q gradient with score-function Q gradient (ablation) | REPPO · DMC and ManiSkill aggregate | aggregate return | Hurt |
| add pathwise gradients through learned Q instead of the PPO GAE clipped objective (ablation) | REPPO · DMC and ManiSkill aggregate | aggregate return | Improved |
| add KL regularization of policy updates (ablation) | REPPO · DMC design ablation | aggregate return | Improved |
| add HL-Gauss categorical value loss (ablation) | REPPO · DMC design ablation | aggregate return | Improved |
| add layer normalization in the critic (ablation) | REPPO · DMC design ablation | aggregate return | Improved |
| add auxiliary self-predictive loss (ablation) | REPPO · DMC design ablation | aggregate return | Mixed |
| shrink the on-policy batch size (ablation) | REPPO · DMC batch size study | mean aggregate performance | Hurt |
| shrink the replay buffer from default (10,485,760) to 32,768 transitions (ablation) | FastTD3 · DMC locomotion | aggregate performance: 10,485,760 -> 32,768 transitions | Hurt |
| set GAE lambda to 0.65 instead of 0.95 (ablation) | REPPO · Atari-10 | human-normalized score: 0.95 -> 0.65 | Improved |
| use REPPO instead of PPO | on-policy RL · DMC and ManiSkill wall-clock | normalized return: about 33% higher normalized returns | Improved |
Closest papers by meaning
- Near Optimal Policy Optimization via REPS (NeurIPS 2021 · similarity 0.89)
- Bridging the Gap Between Value and Policy Based Reinforcement Learning (NeurIPS 2017 · similarity 0.89)
- Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms (JMLR 2022 · similarity 0.89)
- Ranking Policy Gradient (ICLR 2020 · similarity 0.89)
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.89)
- Deterministic Policy Optimization by Combining Pathwise and Score Function Estimators for Discrete Action Spaces (AAAI 2018 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
