Research map / Reinforcement learning / RL from feedback

Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PSN-RLVR (PSN-GRPO), which perturbs LLM policy parameters before rollout generation to get trajectory-level exploration, corrected with truncated importance sampling and a cheap certainty/semantic-diversity adaptive noise scheduler, improving high-budget pass@k on math benchmarks.

Problem

RLVR tends to reweight existing solution traces with low-diversity rollouts (an exploration ceiling), limiting pass@k gains at large sampling budgets; token-level action noise is temporally uncorrelated and harms long chain-of-thought coherence.

Approach

Inject Gaussian noise into policy parameters (MLP blocks) once per iteration before rollout, so the perturbed policy stays fixed across a trajectory; correct the sampling-update mismatch with truncated importance sampling and set the noise scale with a real-time surrogate combining semantic similarity of two probe rollouts and normalized self-certainty.

Key results

• PSN-GRPO on Qwen2.5-Math-7B raises AIME 24 pass@256 by +8.9 points over best training-time temperature baseline (81.6% vs 72.7%).
• TIS raises average pass@256 from 74.33% to 76.94% for PSN-GRPO.
• Combining PSN with pass@k training improves average pass@256 from 76.37% to 79.12%.

Surprising findings

• Naive PSN slightly hurts pass@1 at low budgets while improving high-budget pass@k, an explicit exploration-exploitation trade-off.
• Variant I (KL-based adaptive noise) degrades performance relative to fixed noise due to feedback lag, while the real-time variant helps.

Limitations

• Evaluated mainly on math benchmarks; effectiveness on code or other verifiable domains is untested.
• Gains are marginal on short-response tasks (e.g., AMC 23), per the authors' own limitation that benefits concentrate on long-horizon reasoning.
• Variant II adds about 8% throughput overhead from two probe generations per query.

Experimental setup

Datasets: NuminaMath GSM8K MATH

Benchmarks: AIME 2024 AIME 2025 AMC 2023 OlympiadBench Minerva Math

Metrics: pass@k semantic diversity operation diversity pass@1

Baselines: GRPO Pass@k training RLVR-Decomposed temperature scaling (training-time) temperature scaling (evaluation-time) PSN Fixed PSN Var-I

Builds on: GRPO PPO Parameter-space noise for exploration (Plappert et al.) Truncated importance sampling SimpleRL-Zoo Pass@k training

Closest papers by meaning

• Not Only Where, But When: Temporal Scheduling for RLVR (NeurIPS 2026 · similarity 0.90)
• Parameter Exploration for RLVR via Variational Learning (NeurIPS 2026 · similarity 0.90)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.89)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.89)
• NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation (NeurIPS 2025 · similarity 0.89)