Research map / Large language models / Reasoning and chain of thought

How Far Can Unsupervised RLVR Scale LLM Training?

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Revisits unsupervised RLVR through intrinsic rewards, showing they sharpen the model's prior and follow a rise-then-fall collapse pattern, proposes Model Collapse Step as a trainability indicator, and points to external rewards (self-verification, generation-verification asymmetry) as a more scalable path.

Problem

Intrinsic rewards used for unsupervised RLVR (majority voting, entropy, self-certainty) report early gains, but their scalability, failure modes, and dependence on the model prior are unclear.

Approach

Unifies intrinsic URLVR rewards as cross-entropy manipulations, shows they all converge to sharpening the initial distribution (geometric convergence to the initial majority answer), and uses the collapse step of intrinsic training as a label-free measure of model prior for RL trainability.

Key results

• Intrinsic majority-voting training matches or exceeds ground-truth training early on AIME/AMC, then collapses (proxy reward rises while validation accuracy falls) across hyperparameter settings.
• Training on 32 or 128 DAPO samples avoids collapse, while subsets of 512 or more consistently reward-hack; DAPO-32 reaches 0.057 KL after 600 steps versus about 2x that for DAPO-512.
• Model Collapse Step correlates strongly with GT Gain across 7 models and uses 5.6x fewer tokens than full RL (1.19B vs 6.66B tokens).

Surprising findings

• Even when nearly all initial majority votes are incorrect (32 filtered samples), small-scale intrinsic training still yields gains on AIME24 and AMC23.
• Self-verification on Countdown shows Reward Accuracy dropping then recovering above 0.5 while ground truth reward keeps rising.

Limitations

• Collapse is shown for a limited set of models and hyperparameter ranges; the theory relies on assumptions (majority stability, effective learning) validated mainly empirically.
• Self-verification evaluated only on Countdown with Qwen3 models; external-reward claims are preliminary.
• Model Collapse Step uses a 1% reward-accuracy threshold and aggressive hyperparameters, with ranking stability shown on only 7 models.

Experimental setup

Datasets: DAPO-17k MATH500 Countdown-Tasks-3to4 AIME 2024 AIME 2025 AMC 2023

Benchmarks: AIME 2024 AIME 2025 AMC 2023 Countdown

Metrics: avg@32 accuracy pass@k reward accuracy label accuracy actor entropy KL divergence from reference

Baselines: TTRL Trajectory-Level Entropy Self-Certainty Probability (RLSC) Oracle supervision (ground-truth RLVR) pass@k

Builds on: TTRL RLIF RENT EM-RL DAPO GRPO

Closest papers by meaning

• The Unlearnability Phenomenon in RLVR for Language Models (ICML 2026 · similarity 0.90)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards (ICLR 2026 · similarity 0.90)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.90)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (NeurIPS 2026 · similarity 0.90)