Research map / Reinforcement learning / Policy gradient and actor-critic
Fast Value Tracking for Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LKTD, a scalable SGMCMC-based Kalman temporal-difference sampler that draws posterior samples of deep Q-network parameters without linearization, with W2 convergence proofs and replay-buffer extension, evaluated on an indoor escape task and four Gym control problems.
Problem
Existing Kalman temporal-difference RL methods (KOVA/EKF, UKF) are computationally and memory inefficient for high-dimensional deep networks and rely on linearization, so they cannot provide scalable uncertainty quantification for value functions.
Approach
Reformulate RL as a state-space model with a prior in the state evolution, apply variance splitting and a pseudo-population size, and update parameters with a Langevinized Ensemble Kalman filter so that the chain is a preconditioned SGLD sampler converging to a posterior-like stationary distribution in O(np) time without covariance storage.
Key results
- On the indoor escape environment, LKTD yields notably higher Q-value estimation accuracy than DQN, BootDQN, QR-DQN and KOVA, and its 95% prediction-interval coverage stays close to nominal across pseudo-population sizes.
- LKTD explores both optimal actions N and E across most grids, whereas DQN sticks to one optimal policy.
- Across four Gym tasks (100 replicates), LKTD consistently surpasses DQN and is slightly better than QR-DQN in training reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LKTD instead of DQN, BootDQN, QR-DQN, or KOVA | Deep Q-network in the indoor escape environment · Indoor escape | Q-value MSE | Helped |
| Use LKTD instead of DQN, BootDQN, or KOVA | Deep Q-network in the indoor escape environment · Indoor escape | 95% prediction-interval coverage | Helped |
| Use LKTD instead of QR-DQN | Deep Q-network in the indoor escape environment · Indoor escape | 95% prediction-interval coverage | Helped |
| Use LKTD instead of DQN | Deep Q-network policy · Indoor escape | optimal-policy exploration | Helped |
| Increase pseudo-population size (ablation) | LKTD · Indoor escape | Q-value estimation accuracy | Helped |
| Use LKTD instead of DQN or QR-DQN | Value-based RL agents · Acrobot-v1, CartPole-v1, LunarLander-v2, MountainCar-v0 | training reward | Helped |
| Use LKTD instead of DQN | Value-based RL agents · Acrobot-v1, CartPole-v1, LunarLander-v2, MountainCar-v0 | evaluation reward | Helped |
Limitations
- Convergence theory relies on strong assumptions (dissipativity, Lipschitz gradient, stationary environment) and the replay-buffer extension is only a bias/MSE bound.
- Experiments are limited to a small tabular-like indoor escape task and four classic control environments, with no large-scale benchmarks.
- Pseudo-population size and other hyperparameters (K=5, alpha=0.9) are set heuristically and affect the trade-off between optimization and sampling.
Experimental setup
Datasets: indoor escape environment (custom 100-grid) Acrobot-v1 CartPole-v1 LunarLander-v2 MountainCar-v0
Benchmarks: indoor escape environment Acrobot-v1 CartPole-v1 LunarLander-v2 MountainCar-v0
Metrics: mean squared error of Q-values 95% prediction interval coverage rate mean policy probability cumulative training reward evaluation reward computation time per update
Baselines: DQN BootDQN QR-DQN KOVA
Builds on: Kalman Temporal Difference (KTD) KOVA Langevinized Ensemble Kalman Filter SGLD Bootstrapped DQN
Closest papers by meaning
- Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo (ICLR 2024 · similarity 0.89)
- Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.87)
- Mean Field Langevin Actor-Critic: Faster Convergence and Global Optimality beyond Lazy Learning (ICML 2024 · similarity 0.87)
- Towards Global Optimality for Practical Average Reward Reinforcement Learning without Mixing Time Oracles (ICML 2024 · similarity 0.86)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.86)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.