Research map / Large language models / Reasoning and chain of thought
Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoVo, a self-rewarding RL method that scores reasoning trajectories by the consistency and volatility of intermediate-state likelihoods toward candidate answers (vector-space aggregation) plus a curiosity bonus, training Reinforce++ on unlabeled math instructions.
Problem
RL for LLM reasoning usually needs external labels or reward models; existing self-rewarding methods (majority voting, answer clustering) use only final answers and are prone to reward hacking.
Approach
Uses the likelihood-based distance from each intermediate reasoning state to every candidate final answer to compute consistency (how often states favor their own answer) and volatility (how late states deviate), aggregates them as vectors in polar form whose group magnitude is the reward, and adds a token-probability curiosity bonus.
Key results
- On Llama3.2-3B-Instruct, CoVo reaches 51.2% on MATH-500 vs 51.8% for supervised GRPO, and is best among methods on MMLU-Pro (37.2) and GPQA (32.1).
- On Qwen2.5-3B-Instruct, CoVo reaches 68.2% on MATH-500, above RLOO and GRPO (66.8, 67.4).
- On Qwen2.5-7B-Instruct, CoVo gets 78.4% on MATH-500 and 60.0% on AMC-23, slightly below or matching supervised baselines on several other benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with CoVo instead of few-shot CoT | Qwen2.5-3B-Instruct · MATH-500 | accuracy: 65.4 -> 68.2 | Helped |
| Train with CoVo instead of GRPO | Llama3.2-3B-Instruct · MATH-500 | accuracy: 51.8 -> 51.2 | Hurt |
| Use CoVo instead of the strongest baseline | Qwen2.5-3B-Instruct · AMC-23 | accuracy: 45.0 -> 47.5 | Helped |
| Use CoVo instead of the strongest baseline | Qwen2.5-7B-Instruct · AMC-23 | accuracy: 60.0 -> 60.0 | No clear effect |
| Add curiosity reward to vectorial intrinsic reward (ablation) | CoVo with vectorial intrinsic reward · MATH-500 | accuracy: 51.6 -> 51.2; 68.0 -> 68.2 | Hurt |
| Add curiosity reward to linear intrinsic reward (ablation) | CoVo with linear intrinsic reward · MATH-500 | accuracy: 50.8 -> 51.0; 66.4 -> 67.8 | Mixed |
| Replace linear intrinsic reward with vectorial intrinsic reward, retaining curiosity (ablation) | CoVo · MATH-500 | accuracy: 51.0 -> 51.2; 67.8 -> 68.2 | Mixed |
| Add vectorial intrinsic reward to curiosity reward (ablation) | Curiosity-only variant · MATH-500 | accuracy: 46.2 -> 51.2; 61.8 -> 68.2 | Helped |
| Add linear intrinsic reward to curiosity reward (ablation) | Curiosity-only variant · MATH-500 | accuracy: 46.2 -> 51.0; 61.8 -> 67.8 | Helped |
| Use vectorial rather than linear intrinsic reward with curiosity (ablation) | CoVo · GPQA | accuracy: 31.5 -> 32.1; 31.1 -> 31.7 | Mixed |
Surprising findings
- Ablation shows the curiosity reward alone performs far worse than any intrinsic reward combination, and the vector-aggregation intrinsic reward alone is sometimes better than the linear one on math and science.
- CoVo is reported to match or exceed supervised RL with rule-based rewards on several benchmarks despite using no labels.
Limitations
- Generalizability to vision-language models is untested; authors list it as future work.
- Semi-supervised settings with limited labels are not explored.
- Results are reported as Pass@1 at temperature 0 while training samples at temperature 1.0, and gains over supervised RL are small or mixed.
Experimental setup
Datasets: Open-Reasoner-Zero
Benchmarks: MATH-500 GSM8K AMC-23 Olympiad Bench MMLU-Pro GPQA CommonsenseQA
Metrics: Pass@1 accuracy ACS Entropy Self-BLEU
Baselines: Zeroshot-CoT Fewshot-CoT CoT+SC@4 GRPO RLOO IRPO Reinforce Reinforce++ EMPO TTRL
Builds on: GRPO Reinforce++ TTRL EMPO OpenRLHF
Closest papers by meaning
- Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.93)
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
- Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards (ICLR 2026 · similarity 0.91)
- Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling (NeurIPS 2025 · similarity 0.91)
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.91)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.