Research map / Reinforcement learning / Policy gradient and actor-critic

Stabilizing Policy Optimization via Logits Convexity

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Variance-reduced policy gradients · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Logits Convex Optimization (LCO), which reformulates RL policy optimization as regressing/matching policy logits or distribution to an advantage-shifted optimal target, arguing this restores logits-level convexity and stable gradients relative to PPO.

Problem

RL optimization for LLMs (e.g., PPO) is notoriously unstable compared to SFT, with gradient explosions and training collapse; the causes are poorly understood.

Approach

Identify logits convexity (positive semi-definite Hessian of the loss w.r.t. logits) as the property that makes SFT stable and PPO's clipped objective lacks; then recast RL as fitting the policy to the closed-form optimal policy/logits pi* ∝ pi_old exp(A/β) using MSE, log-cosh, or forward-KL losses, which are logits convex.

Key results

• On Qwen-3-4B, LCO-KLD reaches 73.20% Pass@1 on MATH500 vs. 67.80% for PPO and 67.60% for GRPO.
• On QA-Feedback with Llama-3.2-3B, LCO-KLD averages 0.607 vs. PPO 0.503 and DPO RM 0.552.
• On AlpacaEval 2.0 with Qwen-3-4B, LCO-KLD gets 29.05% WR / 32.93% LC WR vs. PPO 26.79% / 27.20%.

Surprising findings

• LCO policies exceed the DPO-trained reward model used as the advantage source on several math benchmarks despite its larger size.
• PPO gradient norms grow as training loss decreases, with updates dominated by negative-advantage samples at intermediate probabilities.

Limitations

• Convexity analysis is local and relies on a first-order Taylor assumption that the paper itself states is approximate.
• Gains on some benchmarks are within overlapping standard errors (e.g., Qwen-2.5-3B AMC23 and Mistral AlpacaEval), and LCO-MSE is noted as sensitive to advantage noise.
• Advantage signal relies on a DPO-trained implicit reward model, which requires preference data; performance depends on RM quality.

Experimental setup

Datasets: MATH QA-Feedback UltraFeedback MMLU MMLU-Redux

Benchmarks: MATH500 AMC23 MinervaMath QA-Feedback AlpacaEval 2.0 MMLU MMLU-Redux

Metrics: Pass@1 reward score (relevance, factuality, completeness) win rate length-controlled win rate gradient norm

Baselines: REINFORCE PPO GRPO DAPO GSPO MiniLLM GKD SFT

Builds on: PPO DPO GRPO GKD MiniLLM

Closest papers by meaning

• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.89)
• VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.88)
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (ICML 2024 · similarity 0.88)
• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.88)
• Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex (NeurIPS 2026 · similarity 0.87)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.87)