Research map / Large language models / Reasoning and chain of thought

Learning to Self-Verify Makes Language Models Better Reasoners

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that RL training solely for self-verification improves math generation (with shorter reasoning) and proposes integrating self-verification into generation RL via stage-wise initialization or alternating multi-task training.

Problem

LLMs generate good reasoning but verify their own answers poorly; prior work treats verification as an auxiliary signal, leaving the generation-verification asymmetry under-explored, especially whether learning to verify can improve generation.

Approach

Trains the model with GRPO solely on a binary self-verification task over its own on-policy solutions (with data balancing, filtering, and diversity control); finds this improves generation comparably to generation training with much shorter outputs, and proposes Verify-Init and Verify-Alter multi-task RL schedules combining both objectives.

Key results

• Self-Verify training on Qwen2.5-7B-Instruct reaches 38.4 average accuracy vs 38.9 for Generate while using about 1152 vs 4458 tokens on average.
• Verify-Alter raises Qwen2.5-1.5B-Instruct average accuracy from 20.2 to 22.7 over generation-only training.
• On verification of DeepSeek-R1-Distill-Qwen-7B solutions, Self-Verify lifts Qwen2.5-1.5B-Instruct from 45.58 to 62.31 Acc@8.

Surprising findings

• Training only for self-verification improves generation accuracy to roughly the level of generation-only RL, with far fewer tokens.
• Generation training alone does not improve self-verification ability and can even degrade verification on Qwen2.5-3B-Instruct (-4.63).

Limitations

• Extra computation from additional self-verification objective and inference.
• Only evaluated on models up to 7B and mainly on math benchmarks.
• Multi-task schedules (stage-wise, alternating) are manually designed and heuristic.

Experimental setup

Datasets: DAPO-Math-17K AIME24 AIME25 AMC23 Minerva MATH500 OlympiadBench

Benchmarks: AIME24 AIME25 AMC23 Minerva MATH500 OlympiadBench

Metrics: Acc (Avg@16) average tokens Acc@8 verification accuracy Acc@32 majority voting

Baselines: Generate (generation-only RL) Mixed-Train

Builds on: GRPO RLVR DAPO veRL Mixed-Train

Closest papers by meaning

• Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.92)
• Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.91)
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards (NeurIPS 2025 · similarity 0.91)
• The Future of Facts: Tracing the Factual Generation-Verification Gap (NeurIPS 2026 · similarity 0.90)
• On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.89)
• ReVeal: Self-Evolving Code Agents via Reliable Self-Verification (ICLR 2026 · similarity 0.89)