Research map / Reinforcement learning / Policy gradient and actor-critic
Discerning Temporal Difference Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes discerning TD (DTD(λ)), a TD(λ) variant whose eligibility trace and TD-error updates are reweighted by a nonnegative emphasis function, with convergence analysis for a class of emphasis functions and experiments on tabular/linear random-walk tasks.
Problem
Standard TD(λ) weighs all historical states uniformly, which handles visitation imbalance and noisy rewards poorly when assessing a policy's value function.
Approach
Derives a discerning λ-return from an identity that inserts a positive emphasis function f into the multi-step return weights, yielding a backward-view update where the eligibility trace and TD error are both scaled by f, so emphasis controls how TD error is propagated across states.
Key results
- On the 5-state random walk with three initial distributions, DTD(λ) with inverse-visitation emphasis rebalances updates and achieves lower MSPBE than TD(λ) in the later stages of learning.
- With noisy rewards, emphasis exp(-σ(s)) lets DTD(λ) distinguish noise levels and reduce prediction error versus TD(λ) as reward level becomes non-zero.
- With adaptive absolute-expected-TD-error emphasis, DTD(λ) outperforms ETD(λ), PTD and TD(λ, w(·)) across most of the four additional tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use inverse normalized visitation counts as the emphasis function | DTD(λ) vs. TD(λ) · 5-state random walk with left-skewed, center-elevated, and right-skewed visitation | MSPBE | Helped |
| Use negative exponential reward-noise levels as the emphasis function | DTD(λ) vs. TD(λ) · 10-state random walk with noisy rewards | MSPBE | Helped |
| Use absolute expected TD error as a parameter-dependent emphasis | DTD(λ) · 5-state random walk with tabular, inverted, and dependent features; 13-state Boyan chain | MSPBE | Helped |
| Remove the emphasis factor multiplying the TD error (ablation) | DTD(λ), compared with TD(λ, w(·)) · Adaptive-emphasis prediction tasks | MSPBE | Hurt |
Surprising findings
- ETD(λ) performs poorly, which the authors attribute to high variance of the follow-on trace.
- TD(λ) shows faster early progress than DTD(λ) in some settings before DTD(λ) overtakes it.
Limitations
- Convergence guarantees hold only for a restricted class of emphasis functions satisfying contraction conditions.
- Deep RL extension is only discussed conceptually (DAE, PER connection) without deep RL experiments.
- Emphasis design is manual for predetermined forms; adaptive emphasis uses true dynamics in experiments.
Experimental setup
Datasets: 5-state random walk 10-state noisy-reward chain 13-state Boyan chain
Benchmarks: 5-state random walk 10-state noisy-reward chain 13-state Boyan chain
Metrics: MSPBE
Baselines: TD(λ) ETD(λ) PTD TD(λ, w(·))
Builds on: TD(λ) emphatic TD (ETD) prioritized experience replay generalized advantage estimation
Closest papers by meaning
- Towards Parameter-Free Temporal Difference Learning (ICML 2026 · similarity 0.93)
- The surprising efficiency of temporal difference learning for rare event prediction (NeurIPS 2024 · similarity 0.89)
- An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks (ICML 2024 · similarity 0.89)
- Revisiting a Design Choice in Gradient Temporal Difference Learning (ICLR 2025 · similarity 0.89)
- Temporal Difference Flows (ICML 2025 · similarity 0.88)
- Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.