Research map / Reinforcement learning / Policy gradient and actor-critic
Shallow Updates for Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Bayesian-regularized LSTD-Q last-layer updates instead of vanilla DQN | DQN · Breakout, Space Invaders, Asterix, Qbert, Bowling | maximum average score: 401.20 -> 420.00; 1975.50 -> 3207.44; 6011.67 -> 13704.23; 10595.83 -> 10767.47; 42.40 -> 61.21 | Helped |
| Apply Bayesian-regularized FQI last-layer updates instead of vanilla DQN | DQN · Breakout, Space Invaders, Asterix, Qbert, Bowling | maximum average score: 401.20 -> 438.55; 1975.50 -> 3360.81; 6011.67 -> 13636.81; 10595.83 -> 12981.42; 42.40 -> 75.38 | Helped |
| Apply Bayesian-regularized FQI last-layer updates instead of vanilla DDQN | DDQN · Breakout, Space Invaders, Asterix, Qbert, Bowling | maximum average score: 375.00 -> 397.94; 3154.60 -> 4400.83; 15150.00 -> 16270.45; 14875.00 -> 12727.94; 70.50 -> 80.75 | Mixed |
| Use FQI rather than LSTD-Q as the shallow update | LS-DQN · Atari games | maximum average score: 438.55 vs 420.00; 3360.81 vs 3207.44; 13636.81 vs 13704.23; 12981.42 vs 10767.47; 75.38 vs 61.21 | Mixed |
| Use FQI rather than ADAM for the last-layer update (ablation) | DQN on Breakout · Breakout | score difference from DQN: 21 vs -46 | Helped |
| Add Bayesian prior regularization to the ADAM last-layer update (ablation) | ADAM last-layer update · Breakout | score difference from DQN: -96 -> -46 | Helped |
| Increase the ADAM mini-batch size (ablation) | ADAM last-layer update · Breakout | average score | Helped |
| Use only current-policy samples instead of experience replay for the batch update (ablation) | LS-DQN shallow update · Breakout and Qbert | average score | Hurt |
| Perform multiple policy-improvement iterations on the same dataset (ablation) | LSTD-Q and FQI · Breakout and Qbert | average score | No clear effect |
| Add a Bayesian prior regularizer to the least-squares update (ablation) | LSTD-Q and FQI · Breakout and Qbert | average score | Helped |
| Replace Bayesian prior regularization with an unregularized least-squares update (ablation) | LSTD-Q and FQI · High-dimensional Atari observations | average score | Hurt |
Closest papers by meaning
- Deep Double Q-learning (NeurIPS 2026 · similarity 0.89)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.87)
- Confounding Robust Deep Reinforcement Learning: A Causal Approach (NeurIPS 2025 · similarity 0.87)
- Universal Approximation Theorem of Deep Q-Networks (ICML 2025 · similarity 0.85)
- Stochastic Q-learning for Large Discrete Action Spaces (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.