Research map / Reinforcement learning / Policy gradient and actor-critic

Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adds weight normalization to CrossQ (with batch normalization and re-added target networks) so it scales stably to higher update-to-data ratios on 25 DMC and MyoSuite continuous control tasks without network resets.

Problem

CrossQ, a low-UTD model-free RL method using batch normalization, fails to benefit from or scale with higher update-to-data ratios on harder continuous control tasks (DMC dog/humanoid, MyoSuite), with growing critic weight norms reducing the effective learning rate and causing instability.

Approach

Constrain the first two linear layers of the CrossQ critic/actor to unit L2 norm via weight normalization, which keeps weight norms and the effective learning rate stable as UTD increases, combined with batch normalization's scale invariance and a joint forward pass with target networks.

Key results

• CrossQ + WN at UTD=5 uses half the UTD of BRO and no parameter resets, while using about 600k parameters versus about 5M for BRO.
• Vanilla CrossQ at UTD=5 fails to converge reliably across environments, whereas CrossQ + WN scales stably with increasing UTD.
• Removing weight normalization gives the largest performance drop among ablations, especially on MyoSuite; hard weight norm constraint outperforms soft L2 penalty across regularization scales.

Measured effects reported in the paper

Surprising findings

• Vanilla CrossQ does not benefit from higher UTD ratios, and its performance stays similar across UTD values on the tested tasks.
• Parameter resets give only marginal gains for CrossQ + WN, so the method scales without them.

Limitations

• Only continuous state-action benchmarks were evaluated; discrete action or vision-based tasks are unexplored.
• Theory does not connect to convergence rates or sample efficiency guarantees; it addresses empirical phenomena in function approximation.
• Learning rate schedules were not investigated, and hyperparameters were chosen by grid search over learning rate and width.

Experimental setup

Datasets: DeepMind Control Suite MyoSuite

Benchmarks: DeepMind Control Suite MyoSuite

Metrics: interquartile mean return success rate 90% stratified bootstrap confidence intervals environment steps per second

Baselines: BRO CrossQ TD-MPC2 SimBa SR-SAC CrossQ UTD=1 CrossQ UTD=5

Builds on: CrossQ SAC weight normalization BRO SimBa

Closest papers by meaning

• CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity (ICLR 2024 · similarity 0.95)
• Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control (NeurIPS 2024 · similarity 0.87)
• Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data (ICML 2025 · similarity 0.87)
• Compute-Optimal Scaling for Value-Based Deep RL (NeurIPS 2025 · similarity 0.86)
• Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.86)
• Hyperspherical Normalization for Scalable Deep Reinforcement Learning (ICML 2025 · similarity 0.86)