Research map / Reinforcement learning / RL from feedback
Stabilizing Off-policy LLM Optimization with Prefix Importance Ratio
Shiye Lei, Zhihao Cheng, Dacheng Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a prefix importance ratio to stabilize off-policy RL post-training of LLMs, based only on the abstract's framing of correcting sampling-target policy mismatch.
Problem
Off-policy rollouts generated by an older sampling policy create a discrepancy with the current target policy during RL post-training of LLMs, which existing importance-corrected objectives handle imperfectly.
Approach
Uses a prefix-level importance ratio rather than per-token or full-sequence ratios to correct the off-policy mismatch in LLM RL objectives; details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add minimum-prefix ratio factor to token ratio (MinPRO) | CISPO · 7 math benchmarks, n=2, Qwen3-8B-Base | avg pass@1: 56.8 -> 59.4 | Improved |
| add minimum-prefix ratio factor (MinPRO) | CISPO · 7 math benchmarks, n=2, Qwen3-14B-Base | avg pass@1: 61.7 -> 62.6 | Improved |
| use minimum prefix ratio instead of cumulative product (ablation) | MinPRO · n=4, Qwen3-8B-Base | avg pass@1: 52.7 -> 58.3 | Improved |
| use minimum instead of arithmetic mean of prefix ratios (ablation) | MinPRO · n=4, Qwen3-8B-Base | avg pass@1: 54.0 -> 58.3 | Improved |
| use minimum instead of geometric mean of prefix ratios (ablation) | MinPRO · n=4, Qwen3-8B-Base | avg pass@1: 57.3 -> 58.3 | Improved |
| use MinPRO near on-policy (n=0) | MinPRO · Qwen3-8B-Base | avg pass@1: 59.2 (GSPO) vs 59.3 | No clear effect |
| raise staleness from n=4 to n=8 or 16 | MinPRO · Qwen3-8B-Base | avg pass@1: 58.3 -> 57.3 (n=8), 57.2 (n=16) | Hurt |
| use MinPRO against AsymRE, VESPO, SNIB | MinPRO · n=4, Qwen3-8B-Base | avg pass@1: 47.2 / 55.1 / 49.1 -> 58.3 | Improved |
| token-level ratio under off-policy delay (CISPO) | CISPO · Qwen3-8B-Base | training reward and entropy | Hurt |
| use MinPRO on MoE model | MinPRO · Qwen3-30B-A3B-Base | training reward and pass@1 | Improved |
| apply MinPRO at 14B for pass@1 on Minerva | MinPRO · Minerva, Qwen3-14B-Base | pass@1: 33.3 vs CISPO 35.7 | Hurt |
Closest papers by meaning
- VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.92)
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.92)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.91)
- STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.91)
- One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.91)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
