Research map / Reinforcement learning / Policy gradient and actor-critic
Chaining Value Functions for Off-Policy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sequential rather than concurrent chaining (ablation) | Chained TD · Baird-Reward, γ=0.99 | RMSE: 72.6 -> 0.0 | Helped |
| Use sequential rather than concurrent chaining (ablation) | Chained TD · Threestate, γ=0.99 | RMSE: 77.9 -> 0.2 | Helped |
| Use sequential rather than concurrent chaining (ablation) | Chained TD · Baird-Reward, γ=0.9 | RMSE: 0.4 -> 0.0 | Helped |
| Increase the number of chained estimators k (ablation) | Concurrent Chained TD · Baird-Reward and Threestate MDPs | RMSE | Helped |
| Use chained 8-step rather than chained 1-step estimators (ablation) | Chained TD · Threestate, γ=0.99 | RMSE | Helped |
| Normalize the concurrent update gradient (ablation) | Concurrent Chained TD · Baird-Reward MDP | oscillation magnitude | Helped |
| Use chained TD rather than ordinary off-policy TD | Off-policy value prediction · Baird, Baird-Reward, and Threestate MDPs | RMSE / convergence | Helped |
| Use sequential chained TD rather than TDC | Off-policy value prediction · Baird-Reward, γ=0.99 | RMSE: 87.0 -> 0.0 | Helped |
| Use sequential chained TD rather than GTD2 | Off-policy value prediction · Threestate, γ=0.99 | RMSE: 139.6 -> 0.2 | Helped |
| Use concurrent chained TD rather than TDC | Off-policy value prediction · Threestate, γ=0.99 | RMSE: 43.6 -> 77.9 | Hurt |
| Use concurrent chained TD rather than GTD2 | Off-policy value prediction · Baird-Reward, γ=0.99 | RMSE: 83.4 -> 72.6 | Helped |
Closest papers by meaning
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (NeurIPS 2019 · similarity 0.90)
- Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (AAAI 2019 · similarity 0.90)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.89)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.89)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.89)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.