Research map / Reinforcement learning / Policy gradient and actor-critic
Posterior Value Functions: Hindsight Baselines for Policy Gradient Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use posterior value function baseline (hindsight over hidden traffic state) instead of prior value function | REINFORCE advantage estimator on traffic POMDP · Traffic POMDP (Figure 1) | variance of advantage estimator at t=0: from 1 to 0 | Helped |
| Model traffic as stochastic transition in fully observable MDP rather than hidden state (ablation) | posterior advantage estimator · Traffic MDP (Figure 3) | variance of advantage estimator at t=0: from 0 to 1 | Hurt |
| Use posterior value function baseline instead of state value function (on partially observable gridworld) | REINFORCE with belief policy · Partially observable 5x5 gridworld | mean squared error of advantage estimator E[(G_t-B_t)^2] | Helped |
| Use posterior value function baseline instead of prior/observation value function on gridworld | REINFORCE with belief policy returns · Partially observable 5x5 gridworld | episode returns | No clear effect |
| Use posterior value function baseline (theory: never increases variance versus prior/state value function) | policy gradient advantage estimator · All POMDPs (Theorems 1 and 3) | variance of advantage estimator | Helped |
Closest papers by meaning
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.89)
- Provably Efficient Partially Observable Risk-sensitive Reinforcement Learning with Hindsight Observation (ICML 2024 · similarity 0.89)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.88)
- Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction–Reality Gaps (NeurIPS 2026 · similarity 0.88)
- On the Second-Order Convergence of Biased Policy Gradient Algorithms (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.