Research map / Large language models / Reasoning and chain of thought

SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SetPO, which adds a set-level leave-one-out diversity credit (kernelized semantic similarity over rollouts) as an advantage shaping term to GRPO/GSPO/DAPO to preserve output diversity while improving Pass@1 on math reasoning.

Problem

RLVR for LLM reasoning concentrates probability mass on a narrow set of solutions, reducing outcome diversity and entropy, and harming Pass@K.

Approach

Define a set-level diversity objective over sampled trajectories using a kernelized semantic similarity with a diminishing-returns shaping function g(x)=-log(1+x); each trajectory gets a leave-one-out marginal diversity credit that is added to its group-normalized advantage as a plug-in term.

Key results

• SetPO+DAPO on Qwen2.5-Math-7B reaches 53.8% average Pass@1 vs 51.7% for DAPO; SetPO+GRPO raises average from 47.2% to 52.8%.
• On AIME24 with 7B, SetPO+GRPO improves accuracy from 15.7% to 21.6%.
• On Countdown (Qwen2.5-3B, rollout 7), SetPO+GRPO increases diversity width from 351 to 415 and average mode from 2.3 to 3.5.

Surprising findings

• Diversity-promoting advantage shaping improves Pass@1 as well as Pass@K, rather than trading off accuracy.

Limitations

• Diversity is measured with a pretrained embedding model kernel, so results depend on the choice of embedding and similarity function.
• Theoretical monotonicity results rely on assumptions and remainder terms that are not fully controlled in practice.

Experimental setup

Datasets: GSM8K Countdown Simple-RL Zoo hard subset

Benchmarks: GSM8K MATH500 Olympiad College Math AMC23 AIME24 AIME25 Countdown

Metrics: Pass@1 (Avg@64) Pass@K diversity score (1-5) diversity width average mode

Baselines: GRPO GSPO DAPO R1-Zero-Div

Builds on: GRPO DAPO GSPO R1-Zero-Div veRL

Closest papers by meaning

• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.94)
• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity (ICLR 2026 · similarity 0.92)
• Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.90)
• Diversity-Incentivized Exploration for Versatile Reasoning (ICLR 2026 · similarity 0.90)
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.90)