Research map / Reinforcement learning / Policy gradient and actor-critic
Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Soft Resets, which models parameter drift with a learned Ornstein-Uhlenbeck process that softly pulls network parameters toward initialization and raises the effective learning rate, to counter loss of plasticity in non-stationary supervised and off-policy RL.
Problem
Neural networks trained on non-stationary data lose plasticity (failure to fit new data or generalize), and existing remedies such as hard resets discard useful knowledge.
Approach
Model parameter drift as an Ornstein-Uhlenbeck process whose drift parameter gamma is learned online by maximizing approximate predictive likelihood. The drift shrinks parameters toward initialization and increases learning rate in proportion to non-stationarity, giving a soft, data-driven reset instead of a hard one.
Key results
- Bayesian Soft Reset achieves the best overall average online accuracy on plasticity benchmarks, outperforming Online SGD, Hard Reset, L2 Init and Shrink & Perturb.
- Soft Reset outperforms SAC and hard reset on Hopper and Humanoid at high off-policy ratios, with the gap growing as the off-policy ratio increases.
- Learned gamma decreases sharply at task boundaries, and the method is more robust to hyperparameters than the baselines per the sensitivity analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Soft Reset instead of Hard Reset | Online learning on plasticity benchmarks · Permuted MNIST, random-label MNIST, and random-label CIFAR-10 | average per-task online accuracy | Helped |
| Use Soft Reset instead of Online SGD | Online learning on plasticity benchmarks · Data-efficient random-label MNIST and random-label CIFAR-10 | average per-task online accuracy | Helped |
| Use L2 Init instead of Soft Reset | Online learning on plasticity benchmarks · Data-efficient random-label tasks | average per-task online accuracy | Hurt |
| Use Bayesian Soft Reset instead of other Soft Reset variants | Online learning on plasticity benchmarks · Permuted MNIST, random-label MNIST, and random-label CIFAR-10 | average per-task online accuracy | Helped |
| Add 10 drift-parameter updates instead of one (ablation) | Soft Reset · Permuted MNIST, random-label MNIST, and random-label CIFAR-10 | per-task training accuracy | Helped |
| Add 10 neural-network parameter updates (ablation) | Soft Reset with 10 drift-parameter updates · Permuted MNIST, random-label MNIST, and random-label CIFAR-10 | per-task training accuracy | Helped |
| Use per-parameter drift instead of layer-shared drift without Bayesian uncertainty estimates (ablation) | Non-Bayesian Soft Reset variants · Plasticity benchmarks | per-task training accuracy | Hurt |
| Learn uncertainty for neural-network parameters (ablation) | Bayesian Soft Reset · Plasticity benchmarks | per-task training accuracy | Helped |
| Use Soft Reset instead of Online SGD and Hard Reset as task duration increases | Online learning on random-label MNIST with mixed true and random labels · Random-label MNIST with 20%, 40%, or 60% random labels | online accuracy | Helped |
| Use Bayesian Soft Reset instead of Soft Reset (ablation) | Online learning on random-label MNIST with mixed true and random labels · Random-label MNIST with 20%, 40%, or 60% random labels | online accuracy | Helped |
| Use Soft Reset instead of the baselines as off-policy ratio increases | SAC · Hopper-v5 and Humanoid-v4 | average reward | Helped |
| Use the OU drift model with linked mean and variance instead of a drift model with fixed independent variance (ablation) | Soft Reset · Non-stationary learning experiment | performance | Helped |
Surprising findings
- The linearised closed-form loss for gamma was unstable in practice, so the authors resorted to gradient-based estimation.
- Learned gamma mainly changes for the value functions rather than the policy in RL, suggesting non-stationarity mostly comes from the value function.
Limitations
- Code could not be released due to IP constraints, limiting reproducibility.
- Bayesian variant is substantially more computationally and memory expensive than the point-estimate variants.
- Linearisation-based closed-form estimate of gamma was unstable, so the approach relies on stochastic gradient estimates of gamma.
Experimental setup
Datasets: MNIST CIFAR-10 Hopper Humanoid
Benchmarks: permuted MNIST random-label MNIST random-label CIFAR-10 Hopper Humanoid
Metrics: average per-task online accuracy average reward
Baselines: Online SGD Hard Reset L2 Init Shrink & Perturb SAC
Builds on: Shrink & Perturb L2 Init Continual Backprop Bayes by Backprop SAC
Closest papers by meaning
- Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- DARLING: Detection Augmented Reinforcement Learning with Non-Stationary Guarantees (NeurIPS 2026 · similarity 0.85)
- MetaCURL: Non-stationary Concave Utility Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Pausing Policy Learning in Non-stationary Reinforcement Learning (ICML 2024 · similarity 0.85)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.85)
- Constrained Feedback Learning for Non-Stationary Multi-Armed Bandits (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.