Research map / Large language models / Reasoning and chain of thought
Incentivizing LLMs to Self-Verify Their Answers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains LLMs with GRPO to both solve math problems and verify their own solutions (generative self-verification), then aggregates self-verification scores at test time, avoiding external reward models.
Problem
External reward models used for test-time scaling give little gain over majority voting when applied to post-trained reasoning models, due to distribution mismatch between the post-trained generator and general verifiers.
Approach
Unify answer generation and generative verification in a single GRPO run: the verification task reward is based on ground-truth correctness, using a policy-aligned online buffer of recent generations and a dynamic verification reward scaled by GRPO group difficulty; at test time the same model scores its own samples and answers are aggregated with verification-weighted voting.
Key results
- Self-Verification-R1-1.5B reaches 93.60 on MATH500 at 16 samples with test-time scaling, versus 91.00 for self-consistency and 86.00 for best-of-N.
- Self-Verification-Qwen-7B greedy scores 83.60 on MATH500 vs 81.40 for GRPO-Qwen-7B, and its verification accuracy on MATH500 is 87.20 vs GPT-4o 85.20.
- External-RM methods (best-of-N, beam search, DVTS) underperform simple self-consistency for both post-trained models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add verification training to problem-solving RL | Qwen2.5-Math-7B · MATH500 | accuracy: 81.40 -> 83.60 | Helped |
| Add verification training to problem-solving RL | Qwen2.5-Math-7B · AIME24 (avg@10) | accuracy: 19.67 -> 20.00 | Mixed |
| Add verification training to problem-solving RL | Qwen2.5-Math-7B · AIME25 (avg@10) | accuracy | Helped |
| Add verification training to problem-solving RL | DeepSeek-R1-Distill-Qwen-1.5B · AIME24 (avg@10) | accuracy: 37.67 -> 43.00 | Helped |
| Use self-verification instead of self-consistency at 16 samples | Self-Verification-Qwen-7B · MATH500 | accuracy: 86.00 -> 87.20 | Helped |
| Use self-verification instead of self-consistency at 16 samples | Self-Verification-R1-1.5B · MATH500 | accuracy: 91.00 -> 93.60 | Helped |
| Add self-verification training for solution judging | Qwen2.5-Math-7B · MATH500 solution verification | accuracy: 73.20 -> 87.20 | Helped |
| Set aggregation coefficient alpha to 1.0 (ablation) | Self-Verification-Qwen-7B@16 · MATH500 | accuracy: 86.60 -> 87.20 | Helped |
| Increase aggregation coefficient alpha from 1.0 to 3.0 (ablation) | Self-Verification-Qwen-7B@16 · MATH500 | accuracy: 87.20 -> 87.10 | No clear effect |
| Set aggregation coefficient alpha to 0.10 (ablation) | Self-Verification-R1-1.5B@16 · MATH500 | accuracy: 92.60 -> 93.60 | Helped |
| Increase aggregation coefficient alpha from 0.10 to 1.00 (ablation) | Self-Verification-R1-1.5B@16 · MATH500 | accuracy: 93.60 -> 91.00 | Hurt |
| Use a policy-aligned buffer with refresh frequency Tb=20 instead of the default Tb=60 (ablation) | Self-verification RL · Verification training | verification accuracy stability | Hurt |
Surprising findings
- Joint verification training also improved problem-solving accuracy over standard GRPO-only training.
- Simple self-consistency majority voting outperforms external-PRM-guided methods on post-trained models.
Limitations
- Only evaluated on mathematical reasoning; generalization to code and agentic tasks untested.
- Test-time scaling aggregates answers across samples but does not explore response-length scaling or multi-turn verify-revise loops.
- Sensitivity to aggregation coefficient alpha for the smaller 1.5B model; gains from dynamic reward less pronounced for 1.5B.
Experimental setup
Datasets: MATH level 3-5 (SimpleRL-Zoo curated) DeepScaleR math dataset MATH500 AIME24 AIME25 AMC23 OlympiadBench
Benchmarks: MATH500 AIME24 AIME25 AMC23 OlympiadBench
Metrics: accuracy avg@10 accuracy verification accuracy F1 score Pearson correlation
Baselines: GRPO-Qwen-7B SimpleRL-Qwen-Math-7B GRPO-R1-1.5B DeepScaleR-1.5B-Preview Self-consistency Best-of-N Beam Search DVTS
Builds on: GRPO DeepSeek-R1 generative verifiers self-consistency DeepScaleR SimpleRL-Zoo
Closest papers by meaning
- Learning to Self-Verify Makes Language Models Better Reasoners (ICML 2026 · similarity 0.92)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.92)
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.91)
- Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.91)
- Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards (NeurIPS 2025 · similarity 0.91)
- Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.