Research map / Large language models / Reasoning and chain of thought
Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RISE, an online PPO-based RL framework that jointly trains Qwen2.5 models (1.5B/3B/7B) on solution generation and on-policy self-verification, both rewarded by a rule-based outcome verifier.
Problem
LLMs trained with RLVR often show superficial self-reflection: they fail to robustly verify their own outputs, and verification is typically decoupled from solution training.
Approach
Within each RL iteration, the model generates solutions, then critiques its own on-policy solutions via a verification prompt; the outcome verifier's reward is reused as ground truth for the verification score, and generation and verification trajectories are mixed in one PPO batch.
Key results
- RISE-7B reaches 42.9% average reasoning accuracy vs 41.7% for Zero-RL-7B and 41.2% for Instruct-7B, and 69.2% average self-verification accuracy vs 46.6% for Zero-RL-7B.
- RISE-1.5B achieves 74.5% average verification accuracy vs 26.8% for Zero-RL-1.5B, up to 2.8x verification accuracy gain over Zero-RL.
- RISE-7B improves over standard majority voting by +1.9% at k=4 with verification-weighted voting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add online self-verification training to Zero-RL | Qwen2.5-1.5B · Math benchmarks (Avg.) | reasoning accuracy: 24.0 -> 24.4 | Helped |
| Add online self-verification training to Zero-RL | Qwen2.5-1.5B · Math benchmarks (Avg.) | self-verification accuracy: 26.8 -> 74.5; 47.7 percentage point improvement | Helped |
| Add online self-verification training to Zero-RL | Qwen2.5-3B · Math benchmarks (Avg.) | reasoning accuracy: 32.5 -> 33.5 | Helped |
| Add online self-verification training to Zero-RL | Qwen2.5-3B · Math benchmarks (Avg.) | self-verification accuracy: 35.8 -> 74.3 | Helped |
| Add online self-verification training to Zero-RL | Qwen2.5-7B · Math benchmarks (Avg.) | reasoning accuracy: 41.7 -> 42.9 | Helped |
| Add online self-verification training to Zero-RL | Qwen2.5-7B · Math benchmarks (Avg.) | self-verification accuracy: 46.6 -> 69.2 | Helped |
| Use online rather than offline verification data (ablation) | RISE · Math benchmarks | self-verification accuracy | Helped |
| Use online rather than offline verification data (ablation) | RISE · Math benchmarks | problem-solving performance | No clear effect |
| Increase verification batch size from 0 to 100% of the generation batch (ablation) | RISE · Math benchmarks | self-verification performance | Helped |
| Increase verification batch size (ablation) | RISE · Math benchmarks | problem-solving performance | Mixed |
| Use self-verification-weighted voting instead of standard majority voting | RISE-3B and RISE-7B · Math benchmarks, k=4 | accuracy: +0.2% and +1.9% | Helped |
Surprising findings
- Verification reward rises faster and reaches a higher relative level than the problem-solving reward during RL training.
- Offline verification data from a distant policy yields comparable reasoning accuracy but a significant drop in self-verification accuracy.
Limitations
- Evaluated only on mathematical reasoning; generalization to other domains is untested.
- Trains a generative verifier producing natural language critiques; discriminative verifier variant not explored.
- Verification-weighted voting gains are modest (e.g., +0.2% for 3B at k=4).
Experimental setup
Datasets: MATH-Hard (Level 3-5)
Benchmarks: MATH500 Minerva Math OlympiadBench AIME 2024 AMC 2023
Metrics: Pass@1 verification accuracy maj@k verification frequency self-verified reasoning accuracy
Baselines: Zero-RL SFT Qwen2.5-Instruct GPT-4o Math-Shepherd majority voting
Builds on: PPO GAE GRPO verl Zero-RL
Closest papers by meaning
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.91)
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.91)
- Learning to Self-Verify Makes Language Models Better Reasoners (ICML 2026 · similarity 0.91)
- SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- Learning to Reason without External Rewards (ICLR 2026 · similarity 0.90)
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.