Research map / Large language models / Reasoning and chain of thought

You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows RLVR weight trajectories are near rank-1 and linear per tensor, and proposes RELEX, a training-free extrapolation that fits a rank-1 SVD subspace on a short prefix and linearly extrapolates checkpoints to match or exceed full GRPO on math benchmarks.

Problem

Full RLVR training of LLMs is computationally expensive over hundreds of steps; the paper asks whether future RLVR checkpoints can be predicted from early training dynamics.

Approach

Per-tensor SVD of the RLVR weight-delta trajectory yields a shared rank-1 direction whose scalar coefficient evolves near-linearly with training step; RELEX fits a least-squares line to this coefficient on a short prefix and extrapolates the full checkpoint in closed form with no learned model.

Key results

• RELEX with 15% of RLVR cost reaches 71.6% MATH on Qwen2.5-Math-1.5B vs 71.5% for full RLVR, and 85.6% vs 85.5% on Qwen3-4B-Base.
• On Qwen3-8B-Base with 20% cost, RELEX gets 87.4% MATH vs 88.5% for RLVR, and averages 46.2 OOD-avg vs 47.1 for RLVR.
• RELEX extrapolates up to 1000 steps from a 50-125 step prefix; e.g. Qwen2.5-Math-1.5B with Tobs=125 holds 71.6% at step 1000.

Surprising findings

• Projecting onto the rank-1 SVD subspace outperforms extrapolating raw weights, and higher-rank (5, 10) and nonlinear (polynomial, neural) fits do not help.
• Optimal observation window is model-dependent: Qwen3-4B degrades at long horizons with larger windows, while Qwen3-8B benefits from larger windows.

Limitations

• Only studied GRPO on math reasoning with Qwen-family models; generalization to PPO, code, or other families (e.g., Llama) untested.
• Optimal observation window Tobs varies by model and must be validated empirically; long-horizon extrapolation can destabilize.
• Rank-1 design may not generalize universally; adaptive rank selection left open.

Experimental setup

Datasets: MATH

Benchmarks: MATH AIME 2025 AIME 2026 HMMT 2025 OlympiadBench AMC 2023

Metrics: accuracy avg@8 accuracy

Baselines: ExPO AlphaRL Weight Extrapolation Logits Extrapolation RLVR (GRPO) Base

Builds on: GRPO ExPO AlphaRL Weight Extrapolation LoRA

Closest papers by meaning

• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.91)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.91)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning (ICLR 2026 · similarity 0.91)
• Beyond Pass@ 1: Self-Play with Variational Problem Synthesis Sustains RLVR (ICLR 2026 · similarity 0.91)