AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Stabilizing Off-policy LLM Optimization with Prefix Importance Ratio

Shiye Lei, Zhihao Cheng, Dacheng Tao

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 7 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a prefix importance ratio to stabilize off-policy RL post-training of LLMs, based only on the abstract's framing of correcting sampling-target policy mismatch.

Problem

Off-policy rollouts generated by an older sampling policy create a discrepancy with the current target policy during RL post-training of LLMs, which existing importance-corrected objectives handle imperfectly.

Approach

Uses a prefix-level importance ratio rather than per-token or full-sequence ratios to correct the off-policy mismatch in LLM RL objectives; details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add minimum-prefix ratio factor to token ratio (MinPRO)CISPO · 7 math benchmarks, n=2, Qwen3-8B-Baseavg pass@1: 56.8 -> 59.4Improved
add minimum-prefix ratio factor (MinPRO)CISPO · 7 math benchmarks, n=2, Qwen3-14B-Baseavg pass@1: 61.7 -> 62.6Improved
use minimum prefix ratio instead of cumulative product (ablation)MinPRO · n=4, Qwen3-8B-Baseavg pass@1: 52.7 -> 58.3Improved
use minimum instead of arithmetic mean of prefix ratios (ablation)MinPRO · n=4, Qwen3-8B-Baseavg pass@1: 54.0 -> 58.3Improved
use minimum instead of geometric mean of prefix ratios (ablation)MinPRO · n=4, Qwen3-8B-Baseavg pass@1: 57.3 -> 58.3Improved
use MinPRO near on-policy (n=0)MinPRO · Qwen3-8B-Baseavg pass@1: 59.2 (GSPO) vs 59.3No clear effect
raise staleness from n=4 to n=8 or 16MinPRO · Qwen3-8B-Baseavg pass@1: 58.3 -> 57.3 (n=8), 57.2 (n=16)Hurt
use MinPRO against AsymRE, VESPO, SNIBMinPRO · n=4, Qwen3-8B-Baseavg pass@1: 47.2 / 55.1 / 49.1 -> 58.3Improved
token-level ratio under off-policy delay (CISPO)CISPO · Qwen3-8B-Basetraining reward and entropyHurt
use MinPRO on MoE modelMinPRO · Qwen3-30B-A3B-Basetraining reward and pass@1Improved
apply MinPRO at 14B for pass@1 on MinervaMinPRO · Minerva, Qwen3-14B-Basepass@1: 33.3 vs CISPO 35.7Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy