Research map / Large language models / Reasoning and chain of thought

SeRL: Self-play Reinforcement Learning for Large Language Models with Limited Data

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SeRL, a self-play RL framework that bootstraps LLM reasoning from 500 seed math problems by generating and filtering new instructions online and using majority-voting rewards (Reinforce++), matching RL trained on full verifiable MATH data.

Problem

RL for LLM reasoning typically requires large high-quality instruction sets with verifiable rewards, which are scarce in specialized domains; self-instruction methods produce static offline data not suited to online RL and rely on noisy labels.

Approach

Online self-instruction generates new math problems at each training step from a small seed set (with ROUGE-L, keyword, length and dual-end difficulty filters), and rewards are assigned by majority voting over sampled responses instead of ground-truth labels, enabling unsupervised iterative RL.

Key results

• On LLaMA-3.2-3B-Instruct, SeRL iter2 reaches 50.4 on MATH-500 and 72.3 on TabMWP, exceeding RL-GT (49.7 and 69.9) with ground-truth rewards.
• On Qwen-2.5-7B-Instruct after 3 iterations, SeRL outperforms RL-GT on MATH-500 (75.8 vs 74.8), ASDiv and TabMWP.
• Majority-voting reward reaches cosine similarity 0.75 with rule-based reward, versus 0.17 for model-based reward on LLaMA-3.2-3B-Instruct.

Measured effects reported in the paper

Surprising findings

• Without the difficulty filter, the model reward-hacks: MATH-500 accuracy collapses (e.g., 11.6) as it converges on a consistent wrong answer like 0.
• Gains from SFT on the model's own majority or correct responses are negative, while SFT on distilled stronger-model responses helps.

Limitations

• Majority-voting reward is not applicable to tasks without a deterministic final answer, such as writing or proofs.
• Gains plateau after a few iterations (iter 4-6 converge), and the upper bound seems set by the base model's Maj@N.
• Reward reliability degrades on hard problems, mitigated only partially by difficulty filtering.

Experimental setup

Datasets: MATH MedQA

Benchmarks: MATH-500 MATH-Hard ASDiv College Math TabMWP MMLU-Pro MedQA PubMedQA NephSAP

Metrics: Pass@1 cosine similarity MAE MSE

Baselines: RL-GT I-RPO SR-DPO RLOO GRPO SFT on distilled responses RL-MV

Builds on: Reinforce++ Self-Rewarding Language Models Iterative RPO Self-Instruct TTRL

Closest papers by meaning

• SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models (ICLR 2026 · similarity 0.89)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.89)
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning (ICLR 2025 · similarity 0.89)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
• Beyond Pass@ 1: Self-Play with Variational Problem Synthesis Sustains RLVR (ICLR 2026 · similarity 0.89)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.89)