Research map / Reinforcement learning / Policy gradient and actor-critic
Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks
Ryan Sullivan, Akarsh Kumar, Shengyi Huang, John P Dickerson, Joseph Suarez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add all evaluated DreamerV3 tricks | PPO · Arcade Learning Environment, reward clipping | median human-normalized score | No clear effect |
| Add all evaluated DreamerV3 tricks | PPO · Arcade Learning Environment, no reward clipping | median human-normalized score | Improved |
| Add all evaluated DreamerV3 tricks | PPO · DeepMind Control Suite | median normalized return | Hurt |
| Add symlog predictions (ablation) | PPO · Arcade Learning Environment, no reward clipping | median human-normalized score | Improved |
| Add symlog predictions (ablation) | PPO · Arcade Learning Environment, reward clipping | median human-normalized score | Hurt |
| Add twohot encoding (ablation) | PPO · Arcade Learning Environment | median human-normalized score | Hurt |
| Add twohot encoding (ablation) | PPO · DeepMind Control Suite | median normalized return | Hurt |
| Add critic EMA regularizer (ablation) | PPO · DeepMind Control Suite | median normalized return | Hurt |
| Add percentile scaling (ablation) | PPO · DeepMind Control Suite | median normalized return | Improved |
| Add symlog predictions (ablation) | PPO with all tricks · Arcade Learning Environment | median human-normalized score | Improved |
| Add percentile scaling (ablation) | PPO · Atari with reward clipping versus no reward clipping | median human-normalized score | Mixed |
| Add critic EMA regularizer (ablation) | PPO · Atari with reward clipping versus no reward clipping | median human-normalized score | Mixed |
Closest papers by meaning
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.87)
- DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing (ICLR 2024 · similarity 0.87)
- REBEL: Reinforcement Learning via Regressing Relative Rewards (NeurIPS 2024 · similarity 0.87)
- A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization (ICML 2025 · similarity 0.86)
- PPO in the Fisher-Rao geometry (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
