Research map / Large language models / Reasoning and chain of thought
On the Primacy Bias in RLVR Training
Nguyen Phuc, Ngoc-Hieu Nguyen, Parshin Shojaee, Chinh D La, Duy M. H. Nguyen, Rui Zhang, Khoa D Doan, Ting Hua, Nitesh Chawla
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether RLVR can extend LLM reasoning beyond the base model by analyzing primacy bias, the over-reliance on already-learned representations.
Problem
Unclear whether reinforcement learning with verifiable rewards (RLVR) can extend reasoning capabilities beyond what is already encoded in the base model.
Approach
Frames the question through the lens of primacy bias, the tendency of models to over-rely on already-learned representations, applied to RLVR training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train GRPO on hard problems only instead of full dataset (ablation) | GRPO · Qwen2.5-3B-Instruct, Minerva | Pass@128 (%): 48.83 -> 52.98 | Improved |
| train GRPO on hard problems only instead of full dataset (ablation) | GRPO · Qwen2.5-3B-Instruct, OlympiadBench | Pass@128 (%): 67.87 -> 69.87 | Improved |
| train GRPO on easy problems only (ablation) | GRPO · Qwen2.5-3B-Instruct, MATH-500 | Pass@128 (%): 94.31 -> 92.47 | Hurt |
| train GRPO on hard problems only instead of full dataset (ablation) | GRPO · Llama-3.2-3B-Instruct, MATH-500 | Pass@128 (%): 83.4 -> 89.1 | Improved |
| train GRPO on hard problems only vs base (ablation) | GRPO · Qwen2.5-3B-Instruct, MATH-500 | Pass@128 (%): 94.31 -> 94.51 | No clear effect |
| prolong hard-only training from 512 to 2048 steps (ablation) | GRPO (Hard) · Qwen2.5-3B-Instruct, MATH-500 | Pass@k | Hurt |
| prolong hard-only training (ablation) | GRPO (Hard) · Qwen2.5-3B-Instruct, base-unsolvable problems | Pass@k | Improved |
| RL fine-tuning vs SFT | RLVR models · public checkpoints | KL to linearized model: 12.73 vs 0.06 | Improved |
| pre-train then RL vs random initialization (ablation) | GRPO on MNIST · toy MNIST | Pass@k coverage | Hurt |
| increase pre-training skew beta (ablation) | GRPO on MNIST · toy MNIST | steps to learn hard classes | Hurt |
Experimental setup
Builds on: RLVR
Closest papers by meaning
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.92)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.92)
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.92)
- The Unlearnability Phenomenon in RLVR for Language Models (ICML 2026 · similarity 0.92)
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
- On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
