Research map / Large language models / Reasoning and chain of thought
Self-Verifying Reflection Helps Transformers with CoT Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses tiny (1M-16M) transformers trained from scratch on integer multiplication and Sudoku to study self-verifying reflection, with theory on when reflection improves accuracy and analysis of how RL changes verification errors and reflection frequency.
Problem
Weak LLM error detection raises the question of how self-verifying reflection in chain-of-thought actually improves reasoning, and how RL with verifiable rewards changes reflective behavior; natural-language LLMs make this hard to analyze cleanly.
Approach
A minimalistic, language-free reasoning framework where a tiny transformer self-verifies each step with discrete labels and can resample or trace back (reflective MTP and reflective trace-back search), with a theorem giving conditions under which such reflection improves accuracy.
Key results
- Reflective SFT with binary verification substantially improves 1M and 4M models on ID Mult, greatly narrowing the gap across model sizes.
- The best 4M Sudoku model is reported as comparable to OpenAI o3-mini, and the best 16M Mult model outperforms DeepSeek-R1 on ID difficulties.
- GRPO improves ID-Hard accuracy but OOD gains are marginal; RL lowers false negative rate mainly by raising false positive rate rather than improving the verifier.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add binary self-verification during reflective SFT | 1M and 4M transformers · Mult ID-Easy and ID-Hard | accuracy | Helped |
| Add detailed self-verification during reflective SFT | 16M transformer · Mult OOD-Hard | accuracy | Helped |
| Use binary instead of detailed verification in reflective SFT (ablation) | transformers · Mult ID | accuracy | Mixed |
| Use verification targets whose complexity does not match model capacity (ablation) | transformers · Mult | accuracy | Hurt |
| Enable RMTP reflective execution instead of non-reflective execution | SFT transformers · ID-Hard Mult and Sudoku | accuracy | Helped |
| Enable RMTP reflective execution instead of non-reflective execution | 1M transformers · ID-Hard Sudoku | accuracy | No clear effect |
| Use RTBS instead of RMTP (ablation) | transformers · Mult | accuracy | No clear effect |
| Use RTBS instead of RMTP (ablation) | transformers, especially the 4M model with detailed verification · Sudoku | accuracy | Helped |
| Apply GRPO after SFT | 4M and 16M transformers · Mult ID-Hard | accuracy | Helped |
| Apply GRPO after SFT | 4M and 16M transformers · Mult OOD | accuracy | No clear effect |
| Apply GRPO to self-verifying policies | transformers · Mult verification | verification error rates | Mixed |
| Sample GRPO experience CoTs at temperature 1.25 instead of 1 (ablation) | 4M transformer · Mult | reflection frequency | Helped |
Surprising findings
- RL reduces false negatives largely by increasing false positives (an optimistic bias), an error-type trade-off rather than a better verifier.
- Learning to self-verify improves non-reflective forward planning, acting as a regularizer even when reflection is not used at test time.
Limitations
- Generalization to OOD problems remains low and is not improved by RL.
- Findings have not been verified to transfer from tiny transformers to natural-language LLMs.
- Framework is minimalistic and does not exploit natural-language reflection.
Experimental setup
Datasets: synthetic integer multiplication CoTs synthetic Sudoku CoTs
Benchmarks: integer multiplication (ID Easy, ID Hard, OOD Hard) Sudoku (ID Easy, ID Hard, OOD Hard)
Metrics: accuracy false positive rate false negative rate reflection frequency
Baselines: OpenAI o3-mini DeepSeek-R1 non-reflective MTP execution RMTP
Builds on: GRPO PPO RLVR LitGPT o1-journey
Closest papers by meaning
- On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.89)
- Chain of Thought Empowers Transformers to Solve Inherently Serial Problems (ICLR 2024 · similarity 0.89)
- Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement (AAAI 2026 · similarity 0.89)
- Learning to Self-Verify Makes Language Models Better Reasoners (ICML 2026 · similarity 0.89)
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.88)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.