Research map / Reinforcement learning / Policy gradient and actor-critic
Natural Value Approximators: Learning when to Trust Past Estimates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add natural value estimate G_beta in the critic | A3C · Atari 57, human starts | median normalized score: 68.5% -> 93.5% | Helped |
| Add natural value estimate G_beta in the critic | A3C · Atari 57, no-op starts | median normalized score: 91.6% -> 117.0% | Helped |
| Add natural value estimate G_beta in the critic | A3C · Atari 57, human starts | mean normalized score: 310.4% -> 373.3% | Helped |
| Add natural value estimate G_beta in the critic | A3C · Atari 57, no-op starts | games above 75% human: 31/57 -> 32/57 | Helped |
| Use natural value estimate instead of direct estimate on a toy sparse-reward regression | value network · synthetic sparse-reward regression | RMSE | Helped |
| Add hidden ReLU layer (16, 32, 64 units) to beta network (ablation) | NVA · tuning games | performance | No clear effect |
| Weight the v update by usage (1 - beta) (ablation) | NVA · 6 tuning games | performance | Helped |
| Regress G_beta with an aggregate loss into beta and v jointly instead of semantic separate losses (ablation) | NVA · tuning games | performance | Hurt |
| Train with BPTT through the recursive form instead of feed-forward with reward buffer (ablation) | NVA · Atari, human/no-op starts | median normalized score: 103% -> 78% | Hurt |
| Replace NVA with A3C+LSTM memory baseline | A3C · Atari, human starts | median normalized score: 93.5% -> 81% | Hurt |
| Natural value estimate vs direct estimate | A3C critic · Atari 57 games | value estimation error | Helped |
Closest papers by meaning
- The Value-Improvement Path: Towards Better Representations for Reinforcement Learning (AAAI 2021 · similarity 0.90)
- Learning values across many orders of magnitude (NeurIPS 2016 · similarity 0.90)
- Orchestrated Value Mapping for Reinforcement Learning (ICLR 2022 · similarity 0.89)
- Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction (AAAI 2021 · similarity 0.88)
- Learning the Target Network in Function Space (ICML 2024 · similarity 0.86)
- Calibrated Value-Aware Model Learning with Probabilistic Environment Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.