Research map / Reinforcement learning / Visual and representation RL

Regularized Parameter Uncertainty for Improving Generalization in Reinforcement Learning

CVPR 2024 · Reinforcement learning › Visual and representation RL · approach: Probabilistic and autonomous RL · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SNR PUN, a Bayesian parameter-uncertainty PPO policy whose parameters are regularized by capping each weight's mean-to-std ratio, improving out-of-distribution generalization on mutated control environments and transferring to a physical cartpole.

Problem

RL agents overfit to their training environment and generalize poorly to unseen environment configurations; standard KL-based regularization of parameter-uncertainty networks is hard to apply in noisy RL training.

Approach

Replace the KL-to-prior term in Bayes-by-backprop with a loss that penalizes each parameter's signal-to-noise ratio |mu|/sigma above a maximum threshold, shown to be equivalent to a KL against a family of priors; combined with per-timestep resampling, a deterministic critic, and a larger sigma initialization inside PPO.

Key results

• SNR PUN achieves ATP of 670.48 on CartPoleSwingUp, 421.77 on Acrobot, 750.47 on Quadruped and 310.96 on Walker, outperforming the MLP, dropout, gSDE, entropy, weight decay, VIB and PUN baselines in all environments.
• On the physical cartpole, only SNR PUN successfully transferred the policy; its successful swingup percentage is reported as 43% higher than the state-of-the-art.
• Removing the SNR loss (PUN) leaves 2.12% (CartPole) and 10.2% (Quadruped) of SNR values above the max threshold, while SNR PUN has none.

Measured effects reported in the paper

Surprising findings

• Dropout performs well on Quadruped but not on the other environments, and combining PUN with dropout helps only on Acrobot.

Limitations

• Gains on the physical cartpole are reported only for a single simple task and small margins in average reward.
• Max SNR hyperparameter is environment-sensitive and must be tuned per environment.
• Evaluation is restricted to control tasks with mutated parameters; other domains such as supervised continual learning are not tested.

Experimental setup

Datasets: CartPoleSwingUp Acrobot-v1 Quadruped Walker

Benchmarks: CartPoleSwingUp Acrobot-v1 Quadruped Walker physical cartpole

Metrics: average task performance (ATP) successful swingup percentage

Baselines: MLP dropout gSDE maximum entropy RL weight decay VIB PUN

Builds on: Bayes-by-backprop proximal policy optimization variational information bottleneck gSDE

Closest papers by meaning

• State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.86)
• Parseval Regularization for Continual Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
• Doubly Mild Generalization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
• UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection (NeurIPS 2025 · similarity 0.86)
• Skill-aware Mutual Information Optimisation for Zero-shot Generalisation in Reinforcement Learning (NeurIPS 2024 · similarity 0.86)