AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

Source Traces for Temporal Difference Learning

Silviu Pitis

AAAI 2018 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 13 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Combine source traces with experience replay instead of using one-step TD backups during replay (ablation)TD(0) w/ ER and TD Source-SR w/ ER · 3D Gridworldthousands of steps to target error: 58 -> 41; 161 -> 100; 319 -> 189Improved
Add experience replay to TD(0) (ablation)TD(0) · 3D Gridworldthousands of steps to target error: 154 -> 58; 465 -> 161; 979 -> 319Improved
Use source backups instead of one-step TD backups (ablation)TD Source-SR vs TD(0) · 3D Gridworldthousands of steps to target error: 154 -> 58; 465 -> 158; 979 -> 335Improved
Combine row-wise TD SR updates with column-wise TD Source updates (ablation)TD Source-SR · Random MRP and 3D Gridworldsource-map approximation errorImproved
Anneal lambda linearly from 0.5 to 1 over 25,000 steps (ablation)TD Source · Random MRPsource-map approximation errorHurt
Increase partial source-trace lookahead through larger n or lambda (ablation)Partial source learning · Random MRP and 3D Gridworldvalue estimation errorMixed
Use annealed rather than fixed learning rates (ablation)TD(0), TD(lambda), partial source learning, and full source learning · Random MRP and 3D Gridworldvalue estimation errorImproved
Use triple model learning instead of ideal source traces (ablation)Source-trace value learning · 3D Gridworldfinal error: 1.61 -> 1.45Improved
Use SR-reward decomposition rather than direct value updates (ablation)Source-map-based value learning · 3D Gridworldvalue estimation errorMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy