Research map / Large language models / Reasoning and chain of thought

Reinforcement Learning for Reasoning in Large Language Models with One Training Example

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that RLVR with a single math example (1-shot RLVR) or two examples matches full-set RLVR on Qwen2.5-Math-1.5B and generalizes to other models and PPO, with analysis of post-saturation generalization and the role of policy gradient and entropy loss.

Problem

How few training examples are needed for RLVR to improve LLM mathematical reasoning, and what mechanisms drive the gains from very small RLVR training sets?

Approach

Selects training examples by historical variance of training accuracy from a full-data RLVR run, then shows a single example (e.g., pi_1 or pi_13) under GRPO matches the 1.2k DeepScaleR subset; attributes gains mainly to policy gradient loss rather than grokking-style weight decay effects, and finds entropy loss further helps post-saturation generalization.

Key results

• One example (pi_13 or pi_1) lifts Qwen2.5-Math-1.5B MATH500 from 36.0% to 73.6%, matching the 1.2k DSR-sub (73.6%) and average from 17.6% to 35.7% vs 35.9%.
• Two examples {pi_1, pi_13} reach MATH500 74.8% and average 36.6%, slightly exceeding the full DSR-sub run.
• On Qwen2.5-Math-7B, 1-shot RLVR with pi_1 averages 40.2% vs 42.8% for DSR-sub, and 4-shot reaches 42.5%.

Measured effects reported in the paper

Surprising findings

• Training on a single example can be trained for millions of rollouts without overfitting-induced degradation, with test accuracy still improving after training accuracy saturates (post-saturation generalization).
• Entropy loss alone, with no outcome reward, improves MATH500 on Qwen2.5-Math-1.5B from 36.0 to 63.4, though below the format-reward baseline.

Limitations

• The selection criterion (historical variance) is acknowledged as not necessarily optimal for single-example selection.
• DeepSeek-R1-Distill-Qwen-1.5B shows a larger gap between few-shot and full-set RLVR, and 1-shot training degrades after about 100 steps there.
• Results depend on the chosen example and on evaluation/checkpoint selection on test benchmarks (best checkpoint reported).

Experimental setup

Datasets: DeepScaleR-Preview subset (1209 examples) MATH training set (7500)

Benchmarks: MATH500 AIME 2024 AMC 2023 Minerva Math OlympiadBench AIME 2025 ARC-Easy ARC-Challenge

Metrics: pass@1 avg@8 accuracy

Baselines: DSR-sub full-set RLVR MATH full-set RLVR format-reward RLVR random 16-example RLVR LIMR

Builds on: GRPO PPO verl LIMR DeepScaleR

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.94)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.93)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.93)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.93)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.93)
• Spurious Rewards: Rethinking Training Signals in RLVR (ICML 2026 · similarity 0.93)