Research map / Optimization / Nonconvex and smooth optimization
Reanalysis of Variance Reduced Temporal Difference Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VRTD batch pseudo-gradients with batch size M instead of vanilla TD updates | TD · i.i.d. sampling | variance error: \mathcal{O}(\alpha) -> \mathcal{O}(\alpha/M) | Helped |
| Use VRTD batch pseudo-gradients with batch size M instead of vanilla TD updates | TD · Markovian sampling | variance error: \mathcal{O}(\alpha) -> \mathcal{O}(\alpha/M) | Helped |
| Use VRTD batch pseudo-gradients with batch size M instead of vanilla TD updates | TD · Markovian sampling | bias error: \mathcal{O}(\alpha) -> \mathcal{O}(1/M) | Helped |
| Use VRTD instead of vanilla TD | TD · i.i.d. sampling, sufficiently small condition number | overall complexity: \mathcal{O}\left( \frac{1}{\epsilon\lambda^2_A}\log\big(\frac{1}{\epsilon}\big) \right) -> \mathcal{O}\left( \max\Big\{\ | Helped |
| Use VRTD instead of vanilla TD | TD · Markovian sampling | overall complexity: \mathcal{O}\left( \frac{1}{\epsilon\lambda^2_A}\log^2\big(\frac{1}{\epsilon}\big) \right) -> \mathcal{O}\left( \max\Big\ | Helped |
| Increase batch size M | VRTD · i.i.d. and Markovian sampling | squared error | Helped |
| Use independent fresh inner-loop samples rather than selecting updates from the epoch batch | VRTD with i.i.d. sampling · i.i.d. sampling | bias error | Helped |
| Increase batch size M | VRTD · i.i.d. and Markovian sampling | convergence speed | No clear effect |
| Use Markovian samples rather than i.i.d. samples | VRTD · all tested batch sizes | squared error | Hurt |
| Use VRTD with batch size 5000 and constant stepsize rather than TD with a reduced stepsize | TD on Frozen Lake · Frozen Lake | squared error versus pseudo-gradient computations | Helped |
| Use VRTD with batch size 1000 rather than vanilla TD | TD on Mountain Car · Mountain Car | NEU | Helped |
Closest papers by meaning
- Policy Evaluation for Variance in Average Reward Reinforcement Learning (ICML 2024 · similarity 0.88)
- Statistical Efficiency of Distributional Temporal Difference Learning (NeurIPS 2024 · similarity 0.87)
- Discerning Temporal Difference Learning (AAAI 2024 · similarity 0.87)
- The surprising efficiency of temporal difference learning for rare event prediction (NeurIPS 2024 · similarity 0.87)
- Revisiting a Design Choice in Gradient Temporal Difference Learning (ICLR 2025 · similarity 0.86)
- Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.