Research map / Large language models / Reasoning and chain of thought

Beyond Pass@ 1: Self-Play with Variational Problem Synthesis Sustains RLVR

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SvS, an online self-play RLVR method where the policy synthesizes variational problems from its own correct solutions to underperforming training problems (keeping original reference answers), which maintains policy entropy and improves Pass@k on competition math and code.

Problem

Standard RLVR improves Pass@1 while collapsing policy entropy and plateauing Pass@k on limited training problems, and high-quality verifiable problem sets are scarce.

Approach

Online self-play augmentation: the policy rewrites its own correct solutions to underperforming problems into new variational problems that inherit the original reference answer, filters them by a moderate-difficulty reward shaping, and trains jointly on solving original problems, solving synthetic problems, and synthesizing problems.

Key results

• On Qwen2.5-32B-Instruct trained on DAPO-17k, SvS improves Pass@32 over standard RLVR by +18.3 on AIME24 and +22.8 on AIME25 (70.8 vs 52.5; 65.2 vs 42.4).
• Average Pass@1 gains of about 2.5-2.9 points on MATH-12k across 3B, 8B and 32B models, with 32B on MATH-12k gaining 2.5 absolute points overall.
• SvS on code generation reaches significant improvements with five times fewer training steps than the RLVR baseline while keeping entropy stable.

Surprising findings

• Synthetic problems are judged valid by at least one of two LLM judges in over 80% of cases, and manual inspection of problems flagged invalid found no genuinely incorrect ones (judges often flagged unusual but solvable problems).
• Standard RLVR on Qwen2.5-32B with DAPO-17k lost performance on broad general benchmarks, while SvS avoided this degradation and improved some general tasks like MMLU-Pro.

Limitations

• Reward for synthetic problems is a proxy: synthetic reference answers are not verified, so correctness relies on the policy's own solve rate and LLM-judge checks.
• Gains on open-ended-answer benchmarks were negative for SvS on DAPO-17k alone (integer-only overfitting), requiring extra DeepMath data to recover.
• Synthetic problem validity declines slightly over training, possibly due to drift toward the model's own solution style.

Experimental setup

Datasets: MATH-12k DAPO-Math-17k DeepMath PRIME-RL code data

Benchmarks: GSM8K MATH-500 Minerva Math OlympiadBench Gaokao-2023 AMC23 AIME24 AIME25 BeyondAIME Math24o OlymMATH-en-easy OlymMATH-en-hard

Metrics: Pass@1 Pass@32 Pass@k (up to 1024) avg@32

Baselines: standard RLVR (GRPO) SimpleRL Open-Reasoner-Zero Clip-Cov Eup Ext SvS-Asp

Builds on: GRPO Clip-Higher (DAPO) RLVR SwS

Closest papers by meaning

• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.92)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.92)
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.92)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.91)