Research map / Reinforcement learning / Policy gradient and actor-critic
Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Action Value Gradient (AVG), an incremental deep policy gradient method using reparameterization gradients with normalization and TD-error scaling, requiring no replay buffer, batch updates, or target networks, and shows effective learning on simulated benchmarks and real robots.
Problem
Deep policy gradient methods (PPO, SAC, TD3) fail when restricted to small replay buffers or incremental (streaming, single-sample) updates, which limits deployment on resource-constrained onboard robots.
Approach
An incremental on-policy actor-critic that reuses the same reparameterized action for actor and critic updates, with no target networks or replay, stabilized by observation normalization, penultimate-layer feature normalization, and scaling of TD errors by a return-based sigma.
Key results
- AVG is the only incremental method that learns effectively on Gymnasium and DMC tasks, reaching performance comparable to batch SAC (1M buffer) on Gymnasium and surpassing it on Dog tasks.
- Removing normalization and scaling from AVG causes failure; the combined norm_obs, pnorm and scaled_td variant achieves the best overall performance in the ablation.
- On real robots, AVG learns UR-Reacher-2 and Create-Mover, while SAC-1 and SAC-100 fail under the same memory limits; AVG update takes about 37ms vs 67ms for SAC-1 on a Jetson Nano.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use incremental AVG instead of SAC-1 | deep RL on Create-Mover · Create-Mover | episodic return | Helped |
| Use incremental AVG instead of SAC-1 or SAC-100 | deep RL on UR-Reacher-2 · UR-Reacher-2 | episodic return | Helped |
| Add observation normalization, penultimate normalization, and TD-error scaling (ablation) | AVG · continuous-control benchmarks | episodic return | Helped |
| Add normalization and scaling (ablation) | IAC · continuous-control benchmarks | area under the learning curve | Helped |
| Add normalization and scaling (ablation) | SAC-1 · continuous-control benchmarks | episodic return | Mixed |
| Add normalization and scaling (ablation) | TD3-1 · continuous-control benchmarks | episodic return | No clear effect |
| Use penultimate normalization instead of layer normalization or RMS normalization (ablation) | AVG · continuous-control benchmarks | episodic return | Helped |
| Use a target Q-network with conventional slow or fixed updates (ablation) | AVG · continuous-control benchmarks | episodic return | Hurt |
| Use large Polyak update coefficients for the target Q-network (ablation) | AVG · continuous-control benchmarks | episodic return | No clear effect |
| Use distribution entropy instead of sample entropy (ablation) | IAC · continuous-control benchmarks | episodic return | Helped |
Surprising findings
- Target networks hinder AVG in the incremental setting; only very aggressive target updates (large tau, effectively no target) perform comparably.
- Incremental SAC and TD3 variants with normalization and scaling remain unstable, with TD3-1+ failing to learn in all environments.
Limitations
- Low sample efficiency compared to batch methods.
- Sensitive to hyperparameter choice; no single configuration works across environments.
- Limited to continuous action spaces and omits discounting in the state distribution.
Experimental setup
Datasets: MuJoCo Gymnasium tasks (Ant-v4, Hopper-v4, HalfCheetah-v4, Humanoid-v4, Walker2d-v4) DeepMind Control Suite (finger_spin, dog_stand, dog_walk, dog_trot, visual reacher) UR-Reacher-2 (real robot) Create-Mover (real robot)
Benchmarks: Gymnasium MuJoCo DeepMind Control Suite UR-Reacher-2 Create-Mover
Metrics: episodic return area under the curve gradient norm
Baselines: IAC SAC-1 TD3-1 PPO SAC SAC-100
Builds on: SAC TD3 incremental one-step actor-critic penultimate normalization reward scaling for TD error
Closest papers by meaning
- Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (NeurIPS 2017 · similarity 0.89)
- Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications (ICLR 2019 · similarity 0.88)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.88)
- Adaptive Batch Size for Safe Policy Gradients (NeurIPS 2017 · similarity 0.88)
- GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.