Research map / Architectures and efficiency / RNNs and normalization

Normalization and effective learning rates in reinforcement learning

NeurIPS 2024 · Architectures and efficiency › RNNs and normalization · approach: Normalization layer theory · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Normalize-and-Project (NaP), which inserts layer normalization before nonlinearities and projects weights to fixed norm so the effective learning rate stays constant, improving plasticity under nonstationarity in synthetic benchmarks and sequential ALE.

Problem

Normalization layers make the effective learning rate decay as parameter norm grows, which in continual and nonstationary RL can shrink the learning rate to near zero too quickly, causing loss of plasticity.

Approach

Combine insertion of normalization layers before nonlinearities with periodic projection of weights onto a fixed-norm sphere (plus a corresponding explicit per-layer/global learning rate update), making the effective learning rate explicit and constant rather than implicit and decaying.

Key results

• On sequential ALE, NaP significantly outperforms a baseline Rainbow with and without layer normalization, performing similarly to a random initialization even after 200M steps.
• In single-task Atari, Rainbow + NaP with a linear LR schedule outperforms standard Rainbow, Rainbow + LayerNorm, and constant-ELR NaP.
• NaP matches baseline on CIFAR-10 (94.64 vs 94.65) and ImageNet-1k (77.26 vs 77.08) and slightly improves 400M transformer language benchmarks.

Measured effects reported in the paper

Surprising findings

• Constraining parameter norm to fix the effective learning rate in Rainbow frequently yields worse Atari performance than unconstrained parameters, implying implicit learning rate decay is beneficial in deep RL.
• Without an explicit learning rate schedule, Rainbow trained with NaP may fail to make any performance improvement.

Limitations

• Constant effective learning rate can harm performance in single-task RL where implicit decay is beneficial; an explicit schedule is needed.
• Per-layer learning rate schedules are left for future work; global learning rate scaling only approximates the implicit dynamics.
• Projecting scale/offset parameters correctly for homogeneous activations requires extra implementation effort; softer regularization is used instead.

Experimental setup

Datasets: CIFAR-10 ImageNet-1k C4 Atari 57 (Arcade Learning Environment) MNIST-style continual random-label CIFAR-10

Benchmarks: CIFAR-10 ImageNet-1k C4 WikiText-103 Lambada PIQA SIQA Pile Arcade Learning Environment (57 games) Sequential ALE

Metrics: top-1 accuracy per-token accuracy human-normalized score (mean/IQM) online accuracy

Baselines: Rainbow Rainbow + LayerNorm Regenerative regularization Shrink and Perturb ReDo L2 regularization

Builds on: Rainbow DQN Zoo layer normalization Regenerative regularization Shrink and Perturb

Closest papers by meaning

• Normalizing Flows are Capable Models for Continuous Control (NeurIPS 2025 · similarity 0.86)
• Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data (ICML 2025 · similarity 0.86)
• On Double Descent in Reinforcement Learning with LSTD and Random Features (ICLR 2024 · similarity 0.85)
• CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity (ICLR 2024 · similarity 0.85)
• On the Weight Dynamics of Deep Normalized Networks (ICML 2024 · similarity 0.85)
• Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization (NeurIPS 2025 · similarity 0.85)