Research map / Large language models / Reasoning and chain of thought

VinePPO: Refining Credit Assignment in RL Training of LLMs

ICML 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that PPO's learned value network gives poor credit assignment on reasoning tasks and proposes VinePPO, which replaces it with unbiased Monte Carlo value estimates from auxiliary rollouts, improving accuracy and wall-clock efficiency on MATH and GSM8K.

Problem

PPO relies on learned value networks for credit assignment in LLM reasoning, but these estimates are often inaccurate, and simpler critic-free methods (GRPO, RLOO, DPO variants) achieve strong results without fine-grained credit assignment. It is unclear whether improving credit assignment would mea

Approach

Compute advantages using unbiased Monte Carlo value estimates obtained by resetting the language environment to each intermediate reasoning state and sampling K auxiliary completions (used only for value estimation), instead of training a separate value network, while keeping all other PPO hyperparameters fixed.

Key results

• VinePPO reaches PPO's peak accuracy with about 3.0x less wall-clock time on RhoMath 1.1B and 1.51x less time on DeepSeekMath 7B (9x and 2.8x fewer steps).
• PPO's value network identifies the top next action near chance for most of training, while VinePPO consistently identifies it with high accuracy.
• VinePPO consistently outperforms PPO, RLOO, GRPO, RestEM and DPO-Positive on MATH and GSM8K Pass@1 accuracy, with the gap widening on the harder MATH dataset.

Measured effects reported in the paper

Surprising findings

• PPO's value network achieves explained variance of 0.7-0.9 yet still barely outperforms random chance at ranking candidate next steps.
• Many advantage values between consecutive reasoning steps are near zero, so most steps carry little credit signal.

Limitations

• Per-iteration training is slower (up to 5x for RhoMath 1.1B and 2x for DeepSeekMath 7B) due to extra MC sampling.
• Reasoning-step segmentation relies on heuristic rules specific to MATH and GSM8K formats.
• Evaluated only on mathematical reasoning datasets with binary rewards.

Experimental setup

Datasets: MATH GSM8K

Benchmarks: MATH GSM8K

Metrics: Pass@1 accuracy wall-clock time explained variance mean absolute error of value predictions top-action identification accuracy

Baselines: PPO RLOO GRPO RestEM DPO-Positive

Builds on: PPO RLOO GRPO DPO-Positive RestEM

Closest papers by meaning

• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
• Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models (NeurIPS 2025 · similarity 0.90)
• Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025 · similarity 0.89)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.88)
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.88)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs (ICLR 2026 · similarity 0.88)