Research map / Large language models / Reasoning and chain of thought

Reinforcing General Reasoning Without Verifiers

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VeriFree, a verifier-free RL method for general reasoning that uses the policy's own likelihood of the reference answer given its reasoning trace as the reward, and trains Qwen3 base models on WebData.

Problem

R1-Zero-style RL with verifiable rewards is limited to domains with rule-based answer checking; model-based verifiers add reliance on a strong verifier, reward hacking risk, and memory overhead, so general reasoning domains remain hard to train.

Approach

Under a unique-answer assumption, the expected verifier reward equals the probability of the reference answer given the reasoning trace, so RL can maximize that probability directly; the gradient is a probability-weighted policy-gradient term on the reasoning plus a reward-weighted SFT term on the reference answer, which has lower variance (Rao-Blackwellization) than the verifier-based estimator.

Key results

• On MMLU-Pro, Qwen3-8B-Base-VeriFree reaches 67.2 average accuracy versus 65.9 for the verifier-based baseline and 66.9 for Qwen3-8B thinking mode.
• On SuperGPQA, Qwen3-8B-Base-VeriFree reaches 38.0 versus 37.1 for the verifier baseline; Qwen3-4B-Base-VeriFree reaches 35.1 versus 34.3.
• Model confidence pi(y*|x,z) correlates with MMLU-Pro accuracy at Spearman rho = 0.82 during training on Qwen3-8B base.

Surprising findings

• Reference-answer probability rewards matched or beat the verifier-based baseline, even though prior verifier-free variational methods (JLB, LaTRO) were reported to underperform R1-Zero.
• Training only on non-math WebData transfers to math benchmarks such as AIME24 and MATH-500 without any math supervision.

Limitations

• Assumes a unique correct answer string; the method ignores answer equivalence, which the equivalence-class ablation shows gives only slight gains when included.
• Evaluation restricted to multiple-choice or verifiable answers for ease of checking.
• Relies on a single reference answer per question from the dataset.

Experimental setup

Datasets: WebData (WebInstruct-derived, filtered with Qwen2.5-72B-Instruct) MATH-12k (equivalence class ablation)

Benchmarks: MMLU-Pro SuperGPQA GPQA MATH-500 OlympiadBench Minerva Math GSM8K AMC AIME24

Metrics: accuracy average response length

Baselines: Verifier (Dr. GRPO with Qwen2.5-Math-1.5B verifier) Qwen3 instruct (thinking and non-thinking) Qwen2.5-7B-SimpleRL-Zoo Qwen2.5-Math-7B-Oat-Zero General-Reasoner-7B Qwen2.5-7B-Instruct

Builds on: DeepSeek-R1-Zero GRPO RLOO Dr. GRPO LaTRO JLB

Closest papers by meaning

• General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.93)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)
• Generative Verifiers: Reward Modeling as Next-Token Prediction (ICLR 2025 · similarity 0.91)
• Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.90)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
• RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning (NeurIPS 2025 · similarity 0.90)