Research map / Reinforcement learning / RL from feedback

VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VESPO, a variational sequence-level soft importance-weight reshaping kernel phi(W)=W^c1 exp(c2(1-W)) for off-policy RL of LLMs, with a variance bound, evaluated on math and code RL under high staleness.

Problem

Off-policy updates from rollout staleness and train-inference mismatch make sequence-level importance sampling high-variance in LLM RL, while existing token-level clipping or length-normalized sequence-level methods are heuristic and biased.

Approach

Frames importance-weight reshaping as a measure change to an implicit proposal Q, derives a closed-form kernel from a KL-regularized objective with a variance (first-moment) constraint, and applies it directly to sequence-level weights without length normalization, with asymmetric hyperparameters for positive and negative advantages.

Key results

• On Qwen3-30B-A3B-Base (gbs/mbs=8), VESPO averages 70.0% across AIME25/AIME24/AMC23/MATH500 vs 59.1% for the best baseline GSPO.
• At staleness N=64 on Qwen3-30B-A3B-Base, VESPO keeps 61.8% average while GRPO reaches 48.6%, GSPO 47.6%, and SAPO collapses to 16.5%.
• On code generation with the same hyperparameters, VESPO averages 63.0 pass@10 across HumanEval+, MBPP+, and LCB v6, best among GRPO, GSPO, SAPO.

Surprising findings

• SAPO, competitive at low staleness, collapses to near-zero downstream accuracy at N>=32 on the MoE model.
• Adding sqrt(T) or T length normalization to VESPO causes instability and collapse, contrary to the common use of length normalization in GSPO.

Limitations

• The (c1, c2) hyperparameters still require user choice and are fixed by empirical selection rather than derived.
• Only verifiable-reward settings (math and code) are evaluated; preference-based or dense rewards are untested.
• Largest model is 30B-scale; frontier-scale behavior is not verified.

Experimental setup

Datasets: DAPO-Math PRIME-RL/Eurus-2-RL-Data

Benchmarks: AIME24 AIME25 AMC23 MATH-500 HumanEval+ MBPP+ LiveCodeBench v6

Metrics: avg@k accuracy pass@10 training reward

Baselines: GRPO GSPO SAPO TOPR CISPO BAPO

Builds on: GRPO GSPO PPO SAPO DAPO

Closest papers by meaning

• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.90)
• ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.89)
• Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.89)
• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.89)