Research map / Reinforcement learning / RL from feedback

Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NFPO, an N-step forward-trace likelihood-ratio correction to the PPO/masked policy gradient surrogate for RLVR, interpolating between local PPO and exact policy gradient, and reports gains over GRPO and DPPO on math reasoning with Qwen3 base models.

Problem

PPO-style RLVR surrogate objectives are local: they omit the future likelihood-ratio correction of the exact policy-improvement identity, inducing structural bias that trust regions only indirectly control, while the full forward trace is unbiased but high-variance.

Approach

Replace the local surrogate's implicit weight of 1 with a truncated forward trace of the next N-1 token likelihood ratios (stop-gradient, clipped), inside a masked policy gradient with a TV-based token mask, giving a bias-variance knob N that interpolates between PPO (N=1) and the exact policy gradient (N=T).

Key results

• NFPO on Qwen3-1.7B-Base averages 30.7% across seven math benchmarks, 16.3 points above GRPO's ~26.4% per the text and 10.0 above DPPO (table reports 0.307 vs 0.264 and 0.279).
• On Qwen3-8B-Base, NFPO averages 51.5% vs 46.1% for GRPO and 50.1% for DPPO.
• Forward trace roughly halves token-level switch frequency (20.18% to 10.08%).

Surprising findings

• Recomputing rollout log-probabilities during training slightly degrades performance.
• Removing the likelihood-ratio clip on rho_t yields similar performance, attributed to the clipping on the forward trace itself.

Limitations

• Variance of the forward trace grows with N, so N must be tuned (N=4 for 1.7B, N=8 for 8B).
• Theoretical bounds rely on a uniform TV trust-region assumption and bounded rewards, which may not hold in practice.
• Comparison with FIPO is confounded by differing trust-region mechanisms.

Experimental setup

Datasets: MATH level 3-5 DAPO-Math-17k

Benchmarks: AIME24 AIME25 AIME26 AMC23 MATH 500 Minerva OlympiadBench

Metrics: pass@1 averaged over 32 responses average accuracy KL divergence switch frequency

Baselines: GRPO DPPO FIPO IcePop

Builds on: PPO GRPO DPPO TRPO performance difference lemma

Closest papers by meaning

• Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards (ICML 2026 · similarity 0.91)
• What are Key Factors for Updates in RL for LLM Reasoning? (NeurIPS 2026 · similarity 0.91)
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning (ICLR 2026 · similarity 0.90)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.89)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.89)