Research map / Reinforcement learning / RL from feedback

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows group-based RLVR policy gradients implicitly project toward a reward-weighted softmax target on the response simplex, and proposes LPO, which performs exact forward/reverse KL listwise projection onto that target.

Problem

Group-based policy gradient methods for RLVR (GRPO, Dr.GRPO, MaxRL) use group-relative advantages but their optimization geometry is only implicit and approximate; the paper asks what happens when target-projection is made explicit on the finite response simplex.

Approach

Views group-based PG as a first-order approximation of reverse-KL projection toward a softmax(R/tau + s) target over K sampled responses, then performs the projection exactly with forward or reverse KL on the response simplex (listwise, zero-sum, bounded gradient coefficients), with the temperature inherited from the baseline method.</reasoning_effort>

Key results

• LPO variants exceed their matched PG baselines in 13/15 settings for Pass@1 (both forward and reverse KL).
• LPO_fwd beats PG on Pass@k in 15/15 settings; LPO_rev in 11/15.
• LPO variants show lower, more stable gradient norms and higher response entropy than PG baselines.

Surprising findings

• A pointwise forward-KL projection onto the same target suffers a severe performance drop, showing the listwise coupling is essential.
• LPO generates longer responses than PG baselines.

Limitations

• Formulation limited to sequence-level projection under outcome rewards.
• Optimal temperature strategy is task-dependent; no single temperature design dominates.
• Off-policy gradient discrepancy from implicit PG grows linearly with drift, and exact projection is evaluated only on limited models and tasks.

Experimental setup

Datasets: Countdown-34 MATH Polaris PRIME code dataset Geometry3k

Benchmarks: Countdown-34 Countdown-4 AIME24 AIME25 AMC23 MATH500 Minerva Math OlympiadBench Geometry3k test

Metrics: Pass@1 Pass@k response entropy gradient norm response length

Baselines: GRPO Dr.GRPO MaxRL pointwise forward KL projection

Builds on: GRPO Dr.GRPO MaxRL RLOO DAPO

Closest papers by meaning

• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.91)
• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (NeurIPS 2026 · similarity 0.90)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
• All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.89)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.89)