Research map / Reinforcement learning / RL from feedback

Parameter Exploration for RLVR via Variational Learning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Perturbed Parameter Policy Optimization (3PO), which samples rollout policies from an IVON variational posterior over LLM weights to add parameter-space exploration to GRPO-style RLVR, with batched (B3PO, M3PO) and chunked (C3PO) variants.

Problem

RLVR for LLMs suffers from insufficient exploration: zero-advantage groups stall learning, and action-space exploration methods (temperature, entropy regularization) only rescale or reweight token distributions and cannot reorder tokens.

Approach

Use IVON variational learning to maintain a Gaussian posterior over weights and sample perturbed policies at rollout time; C3PO splits each GRPO group across N independently sampled weight perturbations and computes advantages over the full group, so group diversity reflects multiple parameter samples.

Key results

• C3PO achieves the best average Pass@1 across six math benchmarks: 44.04 vs 42.99 for GRPO on OLMo-3-7B and 46.88 vs 45.36 for GRPO on Qwen2.5-Math-7B.
• Gains concentrate on harder AIME benchmarks, outperforming GRPO by up to 4.17 points on OLMo-3 and 5.83 points on Qwen2.5-Math.
• On LiveCodeBench-v6, C3PO reaches 15.17 score, the best among compared methods, at near-identical FLOPs per step to GRPO (about 1.5x wall-clock).

Surprising findings

• Isotropic noise of matched magnitude in C3PO gives average 42.12, below GRPO's 42.99, indicating the learned Hessian-scaled noise matters.
• Loading a learned Hessian from SFT gives negligible benefit over constant initialization, and the SFT Hessian is nearly isotropic.

Limitations

• Wall-clock time is about 1.5x GRPO due to systems inefficiency in multi-model sampling in vLLM.
• Compute limits prevented scaling to larger models and more MC samples; multi-seed robustness checked only for GRPO vs C3PO on OLMo-3 with three seeds.
• Choice of lambda is sensitive; C3PO collapses at lambda=1e8 and Qwen2.5-Math is sensitive to small lambda.

Experimental setup

Datasets: Llama-Nemotron Post-Training Dataset DAPO-MATH-17k CodeR1-12K MATH-500 Minerva AIME 2024-26 AMC 2023 LiveCodeBench-v6

Benchmarks: AIME 2024 AIME 2025 AIME 2026 MATH-500 AMC 2023 Minerva LiveCodeBench-v6

Metrics: Pass@1 (unbiased estimator over K=8 samples) standard error zero-advantage groups rescued degenerate rollout rate incorrect rollout rate

Baselines: GRPO GRPO (EntReg) GRPO (Polaris) GRPO (KLCov) AdamW

Builds on: GRPO IVON DAPO verl

Closest papers by meaning

• Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.90)
• Global Optimality for Constrained Exploration via Penalty Regularization (NeurIPS 2026 · similarity 0.88)
• OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.88)
• Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo (ICLR 2024 · similarity 0.88)
• LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.88)
• TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs (NeurIPS 2026 · similarity 0.87)