Research map / Large language models / Reasoning and chain of thought

SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SwS, which identifies a model's persistently failed RL training problems, synthesizes new problems from their concepts, and continues RLVR training on the augmented set, improving Qwen2.5 3B/7B/32B on math benchmarks.

Problem

RLVR needs high-quality, verifiable math problems at the right difficulty, but human-labeled sets are scarce and existing synthetic augmentation ignores the model's own capabilities, producing problems that are too easy or too hard.

Approach

Use a preliminary RL run to find problems the model consistently fails (accuracy never reaches 50% and trend is decreasing), extract and recombine their concepts per category with a budget proportional to category failure rates, then generate, verify, label answers by self-consistency, filter by difficulty, and continue RL on the augmented set.

Key results

• SwS-7B averages 56.7 vs 46.7 for BaseRL-7B (+10.0 average gain across eight benchmarks).
• SwS-32B averages 68.4 vs 60.7 for BaseRL-32B (+7.7), with AIME25 Avg@32 up 7.2 points.
• Weak-teacher labeling (Qwen2.5-Math-7B-Instruct) with difficulty filtering raises labeling accuracy on retained problems from 80.6% to 97.5%.

Measured effects reported in the paper

Surprising findings

• The self-evolving variant (Qwen2.5-14B-Instruct generating and labeling its own problems) gives only modest gains (+1.2 average) and declines on AIME, which the authors attribute to self-preference in quality filtering.
• Weak-teacher answer labels are reliable enough after difficulty filtering, even though the teacher is weaker than the student on MATH-500 in general.

Limitations

• Relies on a strong instruction model (LLaMA-3.3-70B) and a reasoning model (QwQ-32B) for generation and labeling, adding computation and time cost.
• Synthetic problems from open-source instruction models may lack complexity for the hardest benchmarks such as AIME.
• Self-evolving variant is limited by self-preference in quality assessment and difficulty labeling.

Experimental setup

Datasets: MATH-12k DAPO-Math-17k Light-R1 SFT data Big-Math

Benchmarks: GSM8K MATH-500 Minerva Math OlympiadBench Gaokao 2023 AMC23 AIME24 AIME25

Metrics: Pass@1 (Avg@1) Avg@32

Baselines: Qwen2.5-3B-Instruct Qwen2.5-7B-Instruct Qwen2.5-32B-Instruct BaseRL SimpleRL Open-Reasoner PRIME-RL Oat-Zero

Builds on: GRPO DAPO PromptCoT KP-Math MathScale

Closest papers by meaning

• Beyond Pass@ 1: Self-Play with Variational Problem Synthesis Sustains RLVR (ICLR 2026 · similarity 0.92)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.91)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.91)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.91)