Research map / Reinforcement learning / Policy gradient and actor-critic
Separating value functions across time-scales
Joshua Romoff, Peter Henderson, Ahmed Touati, Emma Brunskill, Joelle Pineau, Yann Ollivier
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Decompose PPO value estimation into discount-difference components | PPO · Qbert | return: 11735 -> 13428 | Improved |
| Decompose PPO value estimation into discount-difference components | PPO · MsPacman | return: 1888 -> 2273 | Improved |
| Decompose PPO value estimation into discount-difference components | PPO · Hero | return: 21038 -> 29074 | Improved |
| Decompose PPO value estimation into discount-difference components | PPO · Amidar | return: 575 -> 731 | Improved |
| Decompose PPO value estimation into discount-difference components | PPO · Zaxxon | return: 7366 -> 396 | Hurt |
| Decompose PPO value estimation into discount-difference components | PPO · WizardOfWor | return: 3408 -> 2118 | Hurt |
| Limit each component's λ_z to 1 (ablation) | PPO-TD(λ, Δ) · Zaxxon | return: 396 -> 3291 | Improved |
| Limit each component's λ_z to 1 (ablation) | PPO-TD(λ, Δ) · WizardOfWor | return: 2118 -> 2440 | Improved |
| Use the fixed TD(λ, Δ) configuration rather than limiting λ_z to 1 (ablation) | PPO-TD(λ̂, Δ) · Frostbite | return: 304 -> 292 | No clear effect |
| Use the fixed TD(λ, Δ) configuration rather than limiting λ_z to 1 (ablation) | PPO-TD(λ̂, Δ) · Qbert | return: 13092 -> 13428 | No clear effect |
| Use an augmented network with summed outputs instead of TD(Δ) component targets (ablation) | PPO · Hero | return: 23511 -> 29074 | Hurt |
| Use shorter component-specific multi-step horizons (ablation) | TD(Δ) · 5-state ring MDP | absolute error | Improved |
Closest papers by meaning
- Discerning Temporal Difference Learning (AAAI 2024 · similarity 0.90)
- Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning (AAAI 2020 · similarity 0.90)
- Towards Parameter-Free Temporal Difference Learning (ICML 2026 · similarity 0.90)
- Temporal Difference Learning: Why It Can Be Fast and How It Will Be Faster (ICLR 2025 · similarity 0.89)
- Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
