Research map / Large language models / Reasoning and chain of thought

Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces an uncertainty-consistency metric (offline point-biserial correlation, plus an online advantage-weighted variant) to select RLVR training queries, reaching near full-data GRPO performance with 30% of queries.

Problem

RLVR for math reasoning needs large query budgets and annotation is costly; classic active learning uncertainty and diversity strategies fail to beat random query selection in this setting.

Approach

Selects queries where subjective uncertainty (PPL/entropy of the model) aligns with objective uncertainty (correctness); offline this is measured by point-biserial correlation, and online by a normalized-advantage-weighted score whose negative correlation with the offline metric is proven theoretically.

Key results

• On Qwen2.5-7B with online selection of 30% data, MATH reaches 72.9 vs 73.2 for full data (vs 68.2 random), and GSM8K reaches 91.7 vs 91.5 full (vs 88.1 random).
• Offline r_pb selection on Qwen2.5-7B yields 72.1 on MATH and 90.1 on GSM8K, above random (70.8, 88.6) and classic AL baselines, but below full data.
• Training on the top-hard (highest objective uncertainty) subset yields lower MATH accuracy than r_pb selection for all three models (e.g., 68.3 vs 72.1 on Qwen2.5-7B).

Surprising findings

• Classic active learning strategies (PPL, entropy, K-center, AskLLM, Active Prompt) fail to beat random selection for RLVR, and some (Active Prompt, CAL) underperform it.
• Training on samples with high subjective-objective inconsistency performs worse than random selection of the same size.

Limitations

• Offline r_pb selection still falls short of full-data performance, attributed to estimation bias from shifting output distributions.
• Online selection cannot use highest objective uncertainty since such samples yield zero reward and no gradient.
• Theoretical analysis relies on strong assumptions (gradient orthogonality, bounded gradient norms) supported only by a small empirical check.

Experimental setup

Datasets: MATH GSM8K DAPO-MATH-17K

Benchmarks: MATH GSM8K

Metrics: Pass@1 (greedy decoding) accuracy

Baselines: Full Random PPL Entropy K-center K-means AskLLM Active Prompt CAL

Builds on: GRPO DAPO RLOO REINFORCE++ K-center Active Prompt

Closest papers by meaning

• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.89)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.89)
• Generalization of RLVR Using Causal Reasoning as a Testbed (ICLR 2026 · similarity 0.88)
• What are Key Factors for Updates in RL for LLM Reasoning? (NeurIPS 2026 · similarity 0.88)
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (ICLR 2026 · similarity 0.88)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.88)