Research map / Large language models / Reasoning and chain of thought

Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PPPO, an RLVR method that optimizes only the prefix tokens of LLM outputs, with progressive prefix retention and continuation accumulated rewards, motivated by a "Beginning Lock-in Effect" in LLM reasoning.

Problem

RLVR training spreads optimization uniformly across all generated tokens, wasting effort on low-return tokens and hindering learning from tokens that determine reasoning quality.

Approach

Identifies a Beginning Lock-in Effect (early reasoning constrains later reasoning) and restricts policy gradients to the prefix of each output, progressively enlarging the retained prefix; rewards for prefixes are computed by accumulating correctness over multiple sampled continuations.

Key results

• PPPO achieves the best accuracy in 14 of 15 backbone-benchmark settings; on Qwen3-8B it averages 74.30 vs 69.13 for DAPO-FT.
• Accuracy improvements of up to 18.02% on a single benchmark (AIME'25) over representative RLVR methods, per the paper.
• PPPO reduces generated output tokens by up to 18.35% while maintaining accuracy.

Surprising findings

• Prefixes from incorrect outputs cause up to 27.5% accuracy drop on AIME'24 (DeepSeek-R1-Distill-Qwen-14B), and adding 'wait'/'however' recovers at most 9.2%.

Limitations

• Continuation sampling multiplies rollout cost per prefix (N x G samples), which the paper does not compare in wall-clock terms.
• Evaluated only on Qwen3 models with a 10240-token response cap and a single math-heavy training set.

Experimental setup

Datasets: DAPO-Math-17K AIME'24 GPQA Diamond

Benchmarks: AIME'24 AIME'25 MATH 500 AMC'23 GPQA Diamond

Metrics: avg@32 accuracy average accuracy AAI/POT learning efficiency output token count avg@4 and var@4

Baselines: GRPO DAPO INTUITOR DAPO-FT

Builds on: GRPO DAPO INTUITOR DAPO with Forking Tokens

Closest papers by meaning

• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.92)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.92)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.91)