Research map / Large language models / Reasoning and chain of thought

VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VI-CuRL, a verifier-independent curriculum for GRPO-style RLVR that selects high-confidence prompts (entropy-derived confidence) with an annealed retention schedule, reducing gradient variance and stabilizing training with or without verifiers.

Problem

Verifier-free RL for LLM reasoning suffers from destructive gradient variance and training collapse, while existing curriculum RL methods rely on ground-truth verifier signals.

Approach

Build a prompt-level curriculum from the model's own entropy-based confidence, use an importance-weighted masked GRPO surrogate with an annealed retention rate beta_t -> 1 (quantile thresholding), and provide a bias-variance decomposition showing reduction of action and problem variance.

Key results

• Verifier-free majority vote on Qwen2.5-Math-1.5B: VI-CuRL averages 34.2 Pass@1 vs 0.1 for TTRL across six math suites.
• On DeepSeek-R1-Distill-Qwen-1.5B with RENT entropy reward, VI-CuRL reaches 37.5 average Pass@1 vs 4.8 for EMPO-style baselines' average (EMPO 23.5) and 21.5 for RENT.
• Variance ratios for kept vs full subsets range 0.55-0.80 early in training (beta_t about 0.2), reflecting a 20-45% reduction.

Surprising findings

• Verifier-free majority-vote TTRL collapses to near 0 Pass@1 on Qwen2.5-Math-1.5B, while the same setup with VI-CuRL matches oracle-level performance.
• Reverse-confidence filtering (keeping lowest-confidence prompts) hurts performance relative to no curriculum, while random filtering has negligible effect.

Limitations

• Evaluated only on verifiable reasoning benchmarks; extension to other RL domains is left for future work.
• Theoretical guarantees rely on bounded surrogate assumption and annealing to beta_t -> 1; the bias term is not removed during early curriculum phases.

Experimental setup

Datasets: DeepScaleR WebInstruct-verified

Benchmarks: AIME-2024 AIME-2025 AMC-2023 MATH500 Minerva MATH OlympiadBench MMLU-Pro GPQA-Diamond TheoremQA WebInstruct-Validation

Metrics: Pass@1 Pass@8

Baselines: VCRL AdaRFT SENT No Curriculum TTRL RENT EMPO

Builds on: GRPO PPO VERL TTRL RENT

Closest papers by meaning

• RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning (NeurIPS 2025 · similarity 0.90)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.89)
• Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards (ICML 2026 · similarity 0.89)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)