Research map / Reinforcement learning / Policy gradient and actor-critic

Reward Inflation: A Healthy Stimulus for Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Policy gradients for special settings · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes reward inflation, a slow exponential scaling of rewards during RL training (with an adaptive PD-style controller, Fed), claimed to induce recency weighting and preserve plasticity, evaluated with DQN on 30 ALE games and SAC on 8 MuJoCo tasks.

Problem

Reward magnitude is usually held fixed during RL training, and its temporal modulation is underexplored; the paper asks whether gradually scaling rewards over training can speed adaptation and preserve plasticity (limit dormant neurons).

Approach

Multiply rewards by a slowly and exponentially growing inflation level before storing them in the replay buffer (stored rewards are not rescaled), giving implicit recency weighting in TD updates and sustained gradient norms that reduce dormant neurons; Fed adapts the inflation rate using the ratio of short- to long-term EMAs of the gradient norm.

Key results

• On 30 ALE games with DQN, 2% fixed reward inflation yields an 11.53% HNS improvement over baseline (16 winning tasks vs 2), and Fed (2%) yields 12.59%.
• On 8 MuJoCo tasks with SAC, 2% inflation improves the normalized score by 11.48% over the baseline (NS 91.3 vs 81.9), and Fed raises this to 16.48% (NS 95.4).
• Deflation (-2%) and 10% inflation degrade MuJoCo performance severely (-24.66% and -23.08% NS change), indicating moderate inflation is needed.

Surprising findings

• Higher inflation rates correlate with fewer dormant neurons, yet the highest rate (10%) gives the fewest dormant neurons but the worst overall performance.
• Learning-rate inflation with the same 2% rate degrades MuJoCo performance well below baseline (NS 61.2), unlike reward inflation.

Limitations

• Inflated reward scale may grow to destabilizing magnitudes under long training horizons; only moderate horizons tested.
• Optimal inflation rate is task-dependent with no universal schedule.
• Fed relies on several hyperparameters and a heuristic control rule.

Experimental setup

Datasets: Arcade Learning Environment (30 games) MuJoCo (8 tasks)

Benchmarks: Arcade Learning Environment MuJoCo

Metrics: human-normalized score episode return min-max normalized score dormant neuron rate TD error vs transition age

Baselines: DQN baseline (0% inflation) SAC baseline (0% inflation) learning rate inflation prioritized experience replay reward normalization return normalization recency-weighted sampling

Builds on: DQN Soft Actor-Critic Prioritized experience replay Emphasizing recent experience (ERE) CleanRL

Closest papers by meaning

• PoEM: Predicting New RL Outcomes from Existing Policies (NeurIPS 2026 · similarity 0.86)
• A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.86)
• The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards (AAAI 2025 · similarity 0.86)
• Learning Goal-Conditioned Representations for Language Reward Models (NeurIPS 2024 · similarity 0.85)
• STARC: A General Framework For Quantifying Differences Between Reward Functions (ICLR 2024 · similarity 0.85)
• Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.85)