Research map / Large language models / Reasoning and chain of thought
ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReVISE, a two-stage preference-learning curriculum that teaches LLMs to self-verify with an [eos]/[refine] token and then self-correct, plus a verification-confidence-weighted majority voting at test time.
Problem
LLMs lack intrinsic ability to detect and correct errors in their own reasoning; existing self-correction relies on expensive external verifiers or extensive RL, which is unstable and costly.
Approach
A special [refine] token lets the model decide after its own answer whether to stop or revise. Training is a two-stage curriculum (verify, then correct) using SFT+DPO on preference pairs built from correct and incorrect sampled reasoning paths, with ground truth appended as the refined completion for wrong answers. At inference, votes are weighted by the probability of the [eos] token.
Key results
- On MATH-500 with Llama-3.1-8B, ReVISE reaches 33.6% Maj@1 vs 30.8% for RFT and 27.8% for SFT; with confidence-weighted Maj@5 it reaches 37.6%.
- On MBPP with Llama-3.2-1B, ReVISE achieves 33.1% Pass@1 vs 30.7% for STaR+ and 30.0% for SFT.
- Against SCoRe on Gemma-2-2B for MATH-500, ReVISE gets 23.2% (23.0% for SCoRe) with about 30x lower training cost, and 25.8% after two iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two-stage curriculum learning instead of training without curriculum (ablation) | ReVISE · GSM8K | accuracy (%): 22.6% to 28.1% | Helped |
| Use two-stage curriculum learning instead of no curriculum (ablation) | ReVISE self-verification · GSM8K | AUROC (%): 71% to 76% | Helped |
| Add stage 2 self-correction training after stage 1 (ablation) | ReVISE self-verification · GSM8K | AUROC (%) | Hurt |
| Add DPO preference loss to SFT loss (ablation) | ReVISE · GSM8K | accuracy (%): -10.3% | Helped |
| Use verification-confidence-weighted voting instead of unweighted majority voting (ablation) | ReVISE · GSM8K, 8 sampled candidates | accuracy (%): 33.2% to 33.9% | Helped |
| Use verification-confidence-weighted voting instead of answer-likelihood scoring (ablation) | ReVISE · GSM8K, 8 sampled candidates | accuracy (%): 32.7% to 33.9% | Helped |
| Refine the initial reasoning rather than stop or regenerate a new response (ablation) | ReVISE · GSM8K and MATH | accuracy (%) | Helped |
| Use two additional iterative training cycles (ablation) | ReVISE · MATH-500 | accuracy (%): 23.2 to 25.8 | Helped |
| Use ReVISE instead of DPO training | Llama-3.2-1B · GSM8K | accuracy (%): 22.6 to 28.1 | Helped |
| Use ReVISE instead of DPO training | Llama-3.2-1B · MATH-500 | accuracy (%): 10.8 to 13.4 | Helped |
| Use ReVISE instead of SCoRe | Gemma-2-2B · MATH-500 | accuracy (%): 23.0 to 23.2 | Helped |
| Use confidence-weighted majority voting over K samples | ReVISE · GSM8K, Maj@2 through Maj@64 | accuracy (%) | Helped |
Surprising findings
- Fine-tuning instruction-tuned Llama-3.2-1B-Instruct on GSM8K with SFT, RFT, or STaR+ underperforms zero-shot CoT, while ReVISE does not.
Limitations
- Training on iterative refinement is not explicitly performed; multi-round refinement is only evaluated at test time.
- Stage 2 training slightly degrades self-verification accuracy, suggesting catastrophic forgetting.
- Evaluated on relatively small models (1B, 8B, 2B) and on math and coding benchmarks only.
Experimental setup
Datasets: GSM8K MATH MetaMath GSM240K MBPP
Benchmarks: GSM8K MATH-500 MBPP
Metrics: accuracy Maj@K Pass@1 AUROC
Baselines: Few-shot CoT SFT RFT STaR+ DPO SCoRe V-STaR verifier Zero-shot CoT
Builds on: DPO STaR Rejection fine-tuning V-STaR Self-consistency
Closest papers by meaning
- Learning to Self-Verify Makes Language Models Better Reasoners (ICML 2026 · similarity 0.89)
- Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement (AAAI 2026 · similarity 0.89)
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.88)
- Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards (NeurIPS 2025 · similarity 0.88)
- ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement (ICLR 2025 · similarity 0.88)
- ReVeal: Self-Evolving Code Agents via Reliable Self-Verification (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.