Research map / Learning theory and algorithms / Prediction and decision losses
On the Statistical Benefits of Temporal Difference Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TD (temporal consistency) instead of first-visit Monte Carlo for value estimation | tabular value estimation in finite MRP · Layered MRP W=5, T=120, n=2000 | MSE of value estimate: about 5x smaller MSE | Helped |
| Use TD instead of MC for advantage estimation | advantage estimation A(s,s') · Layered MRP W=5, T=120, n=2000 | MSE of advantage estimate: up to 100x better | Helped |
| Use TD instead of MC when trajectories pool at a common successor (meeting horizon H=2) (ablation) | value and advantage estimation · MRP with meeting horizon H, varying H | MSE ratio TD/MC | Helped |
| Set meeting horizon H equal to full horizon T (no trajectory pooling) (ablation) | TD vs MC · MRP with meeting horizon H, varying H | MSE ratio TD/MC: TD and MC produce the exact same estimates | No clear effect |
| Richer state representation that removes trajectory pooling | TD · conceptual example (checkout page) | estimation efficiency | No clear effect |
Closest papers by meaning
- Towards Parameter-Free Temporal Difference Learning (ICML 2026 · similarity 0.91)
- Temporal Difference Learning: Why It Can Be Fast and How It Will Be Faster (ICLR 2025 · similarity 0.91)
- The surprising efficiency of temporal difference learning for rare event prediction (NeurIPS 2024 · similarity 0.90)
- Discerning Temporal Difference Learning (AAAI 2024 · similarity 0.89)
- An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks (ICML 2024 · similarity 0.87)
- On Double Descent in Reinforcement Learning with LSTD and Random Features (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.