Research map / Large language models / Reasoning and chain of thought

One-Way Policy Optimization for Self-Evolving LLMs

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes One-Way Policy Optimization (OWPO), which uses verifier-determined advantage sign for direction and a clipped reference-policy log-ratio only to rescale token-level update magnitude, plus iterative reference swaps (a "ratchet"), for self-evolving RLVR on math reasoning.

Problem

Token-level constraints toward a fixed reference policy in RLVR (KL penalties, on-policy distillation) can reverse reward-improving updates when the policy tries to surpass the reference, capping performance under sparse binary verifier rewards.

Approach

Decouple direction from magnitude: the advantage sign fixes the update direction, while a stop-gradient clipped weight exp(-delta) from the signed log-ratio to the reference scales the step (amplify inferior deviations, shrink superior ones), and the reference is periodically replaced by the current policy.

Key results

• On Qwen-3-8B-Base, OWPO reaches 51.90 average Pass@1 at best checkpoint vs. 46.40 for MOPD and 45.31 for DAPO across AIME24/AIME25/AMC.
• On Qwen-2.5-Math-7B-Base, OWPO averages 43.91 Pass@1 (best) vs. 41.11 for MOPD.
• With a suboptimal reference (30.0% AIME24), OWPO reaches about 40% in 4 iterations while MOPD needs 6 iterations to saturate near 37%.

Surprising findings

• Symmetric KL-style regularization (Sym-DAPO) underperforms plain DAPO on several settings, consistent with the claimed force-reversal effect.
• Distillation-based methods (OPD, MOPD) plateau near the reference performance despite fast early gains.

Limitations

• Evaluation limited to math and general reasoning with reliable verifiers; coding, tool use, and interactive RLVR not tested.
• Theory is local first-order and does not characterize global behavior of the clipped objective.
• Still depends on reference policy and hyperparameters such as refresh frequency and weight bounds.

Experimental setup

Datasets: DAPO-Math-17k

Benchmarks: AIME24 AIME25 AMC GPQA MMLU-Pro

Metrics: Pass@1 Pass@16 Average@32

Baselines: GRPO DAPO Sym-DAPO OPD MOPD

Builds on: DAPO GRPO On-Policy Distillation MOPD PPO

Closest papers by meaning

• VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.90)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.90)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.90)
• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.90)